{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Linear models, loss functions, gradients, SGD\n",
    "(c) Deniz Yuret, 2019\n",
    "* Objectives: Define, train and visualize a simple model; understand gradients and SGD; learn to use the GPU.\n",
    "* Prerequisites: [Callable objects](https://docs.julialang.org/en/v1/manual/methods/#Function-like-objects-1), [Generator expressions](https://docs.julialang.org/en/v1/manual/arrays/#Generator-Expressions-1), [MNIST](20.mnist.ipynb), [Iterators](25.iterators.ipynb)\n",
    "* New functions: \n",
    "[mnistdata](https://github.com/denizyuret/Knet.jl/blob/master/data/mnist.jl),\n",
    "[accuracy](http://denizyuret.github.io/Knet.jl/latest/reference/#Knet.accuracy), \n",
    "[zeroone](http://denizyuret.github.io/Knet.jl/latest/reference/#Knet.zeroone), \n",
    "[nll](http://denizyuret.github.io/Knet.jl/latest/reference/#Knet.nll), \n",
    "[Param, @diff, value, params, grad](http://denizyuret.github.io/Knet.jl/latest/reference/#AutoGrad),\n",
    "[sgd](http://denizyuret.github.io/Knet.jl/latest/reference/#Knet.sgd),\n",
    "[progress, progress!](http://denizyuret.github.io/Knet.jl/latest/reference/#Knet.progress), \n",
    "[gpu](http://denizyuret.github.io/Knet.jl/latest/reference/#Knet.gpu), \n",
    "[KnetArray](http://denizyuret.github.io/Knet.jl/latest/reference/#Knet.KnetArray), \n",
    "[load](http://denizyuret.github.io/Knet.jl/latest/reference/#Knet.load), \n",
    "[save](http://denizyuret.github.io/Knet.jl/latest/reference/#Knet.save)\n",
    "\n",
    "\n",
    "<img src=\"https://www.oreilly.com/library/view/tensorflow-for-deep/9781491980446/assets/tfdl_0401.png\" alt=\"A linear model\" width=300/> ([image source](https://www.oreilly.com/library/view/tensorflow-for-deep/9781491980446/ch04.html))\n",
    "\n",
    "In Knet, a machine learning model is defined using plain Julia code. A typical model consists of a **prediction** and a **loss** function. The prediction function takes some input, returns the prediction of the model for that input. The loss function measures how bad the prediction is with respect to some desired output. We train a model by adjusting its parameters to reduce the loss.\n",
    "\n",
    "In this section we will implement a simple linear model to classify MNIST digits. The prediction function will return 10 scores for each of the possible labels 0..9 as a linear combination of the pixel values. The loss function will convert these scores to normalized probabilities and return the average -log probability of the correct answers. Minimizing this loss should maximize the scores assigned to correct answers by the model. We will make use of the loss gradient with respect to each parameter, which tells us the direction of the greatest loss increase. We will improve the model by moving the parameters in the opposite direction (using a GPU if available). We will visualize the model weights and performance over time. The final accuracy of about 92% is close to the limit of what we can achieve with this type of model. To improve further we must look beyond linear models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# Set display width, load packages, import symbols\n",
    "ENV[\"COLUMNS\"]=72\n",
    "using Pkg; for p in (\"Knet\",\"AutoGrad\",\"Plots\",\"Images\",\"ImageMagick\"); haskey(Pkg.installed(),p) || Pkg.add(p); end\n",
    "using Statistics: mean\n",
    "using Base.Iterators: flatten\n",
    "import Random # seed!\n",
    "using Knet: Knet, AutoGrad, dir, Data, Param, @diff, value, params, grad, progress, progress!, gpu, KnetArray, load, save\n",
    "# The following are defined for instruction even though they are provided in Knet\n",
    "# using Knet: accuracy, zeroone, nll, sgd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "┌ Info: Loading MNIST...\n",
      "└ @ Main /home/deniz/.julia/dev/Knet/data/mnist.jl:33\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "600-element Data{Tuple{Array{Float32,2},Array{UInt8,1}}}\n",
      "100-element Data{Tuple{Array{Float32,2},Array{UInt8,1}}}\n"
     ]
    }
   ],
   "source": [
    "# Load data (mnistdata basically replicates mnist.ipynb)\n",
    "include(Knet.dir(\"data\",\"mnist.jl\"))\n",
    "dtrn,dtst = mnistdata(xsize=(784,:),xtype=Array)\n",
    "println.(summary.((dtrn,dtst)));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Model definition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Linear([-0.00577016 0.00740097 … 0.000858401 0.0100368; 0.00508172 0.00706544 … 0.000725378 -0.00708007; … ; -0.0163844 -0.0174381 … -0.0148394 -0.00358024; 0.00810655 0.00301301 … 0.011367 -0.00975392], [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# In Julia we define a new datatype using `struct`:\n",
    "struct Linear; w; b; end\n",
    "\n",
    "# The new struct comes with a default constructor:\n",
    "model = Linear(0.01 * randn(10,784), zeros(10))\n",
    "\n",
    "# We can define other constructors with different inputs:\n",
    "Linear(i::Int,o::Int,scale=0.01) = Linear(scale * randn(o,i), zeros(o))\n",
    "\n",
    "# This one allows instances to be defined using input and output sizes:\n",
    "model = Linear(784,10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# We turn Linear instances into callable objects for prediction:\n",
    "(m::Linear)(x) = m.w * x .+ m.b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(\"784×100 Array{Float32,2}\", \"100-element Array{UInt8,1}\")"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x,y = first(dtst) # The first minibatch from the test set\n",
    "summary.((x,y))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1×100 LinearAlgebra.Adjoint{Int64,Array{Int64,1}}:\n",
       " 7  2  1  10  4  1  4  9  5  9  …  1  3  6  9  3  1  4  1  7  6  9"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Int.(y)' # correct answers are given as an array of integers (remember we use 10 for 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10×100 Array{Float64,2}:\n",
       " -0.0573706    0.0836898   -0.0993739   …   0.0341014   -0.125484   \n",
       " -0.0340025    0.0118058    0.0451012      -0.0658874    0.0276873  \n",
       "  0.0293171    0.070559    -0.00201588      0.085163     0.0388435  \n",
       "  0.068166    -0.0514477   -0.0772223      -0.0186688    0.0323772  \n",
       " -0.00441849  -0.025335    -0.0217464       0.0426281   -0.000801449\n",
       "  0.102021     0.0436141    0.0867583   …   0.0987738    0.0433309  \n",
       " -0.0436877    0.0792506   -0.00330623     -0.0109909   -0.113095   \n",
       "  0.167162     0.00859507  -0.0264033       0.0839035    0.17578    \n",
       "  0.0891127   -0.0194897    0.0988181      -0.00689439   0.151461   \n",
       " -0.0394966   -0.0226349   -0.129688        0.0667691    0.117331   "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ypred = model(x)  # Predictions on the first minibatch: a 10x100 score matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.15"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We can calculate the accuracy of our model for the first minibatch\n",
    "accuracy(model,x,y) = mean(y' .== map(i->i[1], findmax(Array(model(x)),dims=1)[2]))\n",
    "accuracy(model,x,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.09829999999999998"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We can calculate the accuracy of our model for the whole test set\n",
    "accuracy(model,data) = mean(accuracy(model,x,y) for (x,y) in data)\n",
    "accuracy(model,dtst)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9017000000000001"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# ZeroOne loss (or error) is defined as 1 - accuracy\n",
    "zeroone(x...) = 1 - accuracy(x...)\n",
    "zeroone(model,dtst)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Loss function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "nll (generic function with 1 method)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# For classification we use negative log likelihood loss (aka cross entropy, softmax loss, NLL)\n",
    "# This is the average -log probability assigned to correct answers by the model\n",
    "function nll(scores, y)\n",
    "    expscores = exp.(scores)\n",
    "    probabilities = expscores ./ sum(expscores, dims=1)\n",
    "    answerprobs = (probabilities[y[i],i] for i in 1:length(y))\n",
    "    mean(-log.(answerprobs))\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.2995940410919387"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# model(x) gives predictions, let model(x,y) give the loss\n",
    "(m::Linear)(x, y) = nll(m(x), y)\n",
    "model(x,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.2995940410919387"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We can also use the Knet nll implementation for efficiency\n",
    "(m::Linear)(x, y) = Knet.nll(m(x), y)\n",
    "model(x,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# If the input is a dataset compute average loss:\n",
    "(m::Linear)(data::Data) = mean(m(x,y) for (x,y) in data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.300518331889146"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Here is per-instance average negative log likelihood for the whole test set\n",
    "model(dtst)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Bonus question:** What is special about the loss value 2.3?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Calculating the gradient using AutoGrad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/latex": [
       "Usage:\n",
       "\n",
       "\\begin{verbatim}\n",
       "x = Param([1,2,3])          # user declares parameters with `Param`\n",
       "x => P([1,2,3])             # `Param` is just a struct wrapping a value\n",
       "value(x) => [1,2,3]         # `value` returns the thing wrapped\n",
       "sum(x .* x) => 14           # Params act like regular values\n",
       "y = @diff sum(x .* x)       # Except when we differentiate using `@diff`\n",
       "y => T(14)                  # you get another struct\n",
       "value(y) => 14              # which carries the same result\n",
       "params(y) => [x]            # and the Params that it depends on \n",
       "grad(y,x) => [2,4,6]        # and the gradients for all Params\n",
       "\\end{verbatim}\n",
       "\\texttt{Param(x)} returns a struct that acts like \\texttt{x} but marks it as a parameter you want to compute gradients with respect to.\n",
       "\n",
       "\\texttt{@diff expr} evaluates an expression and returns a struct that contains the result (which should be a scalar) and gradient information.\n",
       "\n",
       "\\texttt{grad(y, x)} returns the gradient of \\texttt{y} (output by @diff) with respect to any parameter \\texttt{x::Param}, or  \\texttt{nothing} if the gradient is 0.\n",
       "\n",
       "\\texttt{value(x)} returns the value associated with \\texttt{x} if \\texttt{x} is a \\texttt{Param} or the output of \\texttt{@diff}, otherwise returns \\texttt{x}.\n",
       "\n",
       "\\texttt{params(x)} returns an iterator of Params found by a recursive search of object \\texttt{x}.\n",
       "\n",
       "Alternative usage:\n",
       "\n",
       "\\begin{verbatim}\n",
       "x = [1 2 3]\n",
       "f(x) = sum(x .* x)\n",
       "f(x) => 14\n",
       "grad(f)(x) => [2 4 6]\n",
       "gradloss(f)(x) => ([2 4 6], 14)\n",
       "\\end{verbatim}\n",
       "Given a scalar valued function \\texttt{f}, \\texttt{grad(f,argnum=1)} returns another function \\texttt{g} which takes the same inputs as \\texttt{f} and returns the gradient of the output with respect to the argnum'th argument. \\texttt{gradloss} is similar except the resulting function also returns f's output.\n",
       "\n"
      ],
      "text/markdown": [
       "Usage:\n",
       "\n",
       "```\n",
       "x = Param([1,2,3])          # user declares parameters with `Param`\n",
       "x => P([1,2,3])             # `Param` is just a struct wrapping a value\n",
       "value(x) => [1,2,3]         # `value` returns the thing wrapped\n",
       "sum(x .* x) => 14           # Params act like regular values\n",
       "y = @diff sum(x .* x)       # Except when we differentiate using `@diff`\n",
       "y => T(14)                  # you get another struct\n",
       "value(y) => 14              # which carries the same result\n",
       "params(y) => [x]            # and the Params that it depends on \n",
       "grad(y,x) => [2,4,6]        # and the gradients for all Params\n",
       "```\n",
       "\n",
       "`Param(x)` returns a struct that acts like `x` but marks it as a parameter you want to compute gradients with respect to.\n",
       "\n",
       "`@diff expr` evaluates an expression and returns a struct that contains the result (which should be a scalar) and gradient information.\n",
       "\n",
       "`grad(y, x)` returns the gradient of `y` (output by @diff) with respect to any parameter `x::Param`, or  `nothing` if the gradient is 0.\n",
       "\n",
       "`value(x)` returns the value associated with `x` if `x` is a `Param` or the output of `@diff`, otherwise returns `x`.\n",
       "\n",
       "`params(x)` returns an iterator of Params found by a recursive search of object `x`.\n",
       "\n",
       "Alternative usage:\n",
       "\n",
       "```\n",
       "x = [1 2 3]\n",
       "f(x) = sum(x .* x)\n",
       "f(x) => 14\n",
       "grad(f)(x) => [2 4 6]\n",
       "gradloss(f)(x) => ([2 4 6], 14)\n",
       "```\n",
       "\n",
       "Given a scalar valued function `f`, `grad(f,argnum=1)` returns another function `g` which takes the same inputs as `f` and returns the gradient of the output with respect to the argnum'th argument. `gradloss` is similar except the resulting function also returns f's output.\n"
      ],
      "text/plain": [
       "  Usage:\n",
       "\n",
       "\u001b[36m  x = Param([1,2,3])          # user declares parameters with `Param`\u001b[39m\n",
       "\u001b[36m  x => P([1,2,3])             # `Param` is just a struct wrapping a value\u001b[39m\n",
       "\u001b[36m  value(x) => [1,2,3]         # `value` returns the thing wrapped\u001b[39m\n",
       "\u001b[36m  sum(x .* x) => 14           # Params act like regular values\u001b[39m\n",
       "\u001b[36m  y = @diff sum(x .* x)       # Except when we differentiate using `@diff`\u001b[39m\n",
       "\u001b[36m  y => T(14)                  # you get another struct\u001b[39m\n",
       "\u001b[36m  value(y) => 14              # which carries the same result\u001b[39m\n",
       "\u001b[36m  params(y) => [x]            # and the Params that it depends on \u001b[39m\n",
       "\u001b[36m  grad(y,x) => [2,4,6]        # and the gradients for all Params\u001b[39m\n",
       "\n",
       "  \u001b[36mParam(x)\u001b[39m returns a struct that acts like \u001b[36mx\u001b[39m but marks it as a\n",
       "  parameter you want to compute gradients with respect to.\n",
       "\n",
       "  \u001b[36m@diff expr\u001b[39m evaluates an expression and returns a struct that\n",
       "  contains the result (which should be a scalar) and gradient\n",
       "  information.\n",
       "\n",
       "  \u001b[36mgrad(y, x)\u001b[39m returns the gradient of \u001b[36my\u001b[39m (output by @diff) with respect\n",
       "  to any parameter \u001b[36mx::Param\u001b[39m, or \u001b[36mnothing\u001b[39m if the gradient is 0.\n",
       "\n",
       "  \u001b[36mvalue(x)\u001b[39m returns the value associated with \u001b[36mx\u001b[39m if \u001b[36mx\u001b[39m is a \u001b[36mParam\u001b[39m or the\n",
       "  output of \u001b[36m@diff\u001b[39m, otherwise returns \u001b[36mx\u001b[39m.\n",
       "\n",
       "  \u001b[36mparams(x)\u001b[39m returns an iterator of Params found by a recursive search\n",
       "  of object \u001b[36mx\u001b[39m.\n",
       "\n",
       "  Alternative usage:\n",
       "\n",
       "\u001b[36m  x = [1 2 3]\u001b[39m\n",
       "\u001b[36m  f(x) = sum(x .* x)\u001b[39m\n",
       "\u001b[36m  f(x) => 14\u001b[39m\n",
       "\u001b[36m  grad(f)(x) => [2 4 6]\u001b[39m\n",
       "\u001b[36m  gradloss(f)(x) => ([2 4 6], 14)\u001b[39m\n",
       "\n",
       "  Given a scalar valued function \u001b[36mf\u001b[39m, \u001b[36mgrad(f,argnum=1)\u001b[39m returns another\n",
       "  function \u001b[36mg\u001b[39m which takes the same inputs as \u001b[36mf\u001b[39m and returns the gradient\n",
       "  of the output with respect to the argnum'th argument. \u001b[36mgradloss\u001b[39m is\n",
       "  similar except the resulting function also returns f's output."
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "@doc AutoGrad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Linear"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Redefine the constructor to use Param's so we can compute gradients\n",
    "Linear(i::Int,o::Int,scale=0.01) = \n",
    "    Linear(Param(scale * randn(o,i)), Param(zeros(o)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Set random seed for replicability\n",
    "Random.seed!(9);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Linear(P(Array{Float64,2}(10,784)), P(Array{Float64,1}(10)))"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Use a larger scale to get a large initial loss\n",
    "model = Linear(784,10,1.0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "19.10423456298375"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We can still do predictions and calculate loss:\n",
    "model(x,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "T(19.10423456298375)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# And we can do the same loss calculation also computing gradients:\n",
    "J = @diff model(x,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "19.10423456298375"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# To get the actual loss value from J:\n",
    "value(J)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2-element Array{Param,1}:\n",
       " P(Array{Float64,1}(10))    \n",
       " P(Array{Float64,2}(10,784))"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# params(J) returns an iterator of Params J depends on (i.e. model.b, model.w):\n",
    "params(J) |> collect"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10×784 Array{Float64,2}:\n",
       " 0.0  0.0  0.0  0.0  0.0  0.0  0.0  …  0.0  0.0  0.0  0.0  0.0  0.0\n",
       " 0.0  0.0  0.0  0.0  0.0  0.0  0.0     0.0  0.0  0.0  0.0  0.0  0.0\n",
       " 0.0  0.0  0.0  0.0  0.0  0.0  0.0     0.0  0.0  0.0  0.0  0.0  0.0\n",
       " 0.0  0.0  0.0  0.0  0.0  0.0  0.0     0.0  0.0  0.0  0.0  0.0  0.0\n",
       " 0.0  0.0  0.0  0.0  0.0  0.0  0.0     0.0  0.0  0.0  0.0  0.0  0.0\n",
       " 0.0  0.0  0.0  0.0  0.0  0.0  0.0  …  0.0  0.0  0.0  0.0  0.0  0.0\n",
       " 0.0  0.0  0.0  0.0  0.0  0.0  0.0     0.0  0.0  0.0  0.0  0.0  0.0\n",
       " 0.0  0.0  0.0  0.0  0.0  0.0  0.0     0.0  0.0  0.0  0.0  0.0  0.0\n",
       " 0.0  0.0  0.0  0.0  0.0  0.0  0.0     0.0  0.0  0.0  0.0  0.0  0.0\n",
       " 0.0  0.0  0.0  0.0  0.0  0.0  0.0     0.0  0.0  0.0  0.0  0.0  0.0"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# To get the gradient of a parameter from J:\n",
    "∇w = grad(J,model.w)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "∇b = grad(J, model.b) = [-0.139954, -0.064541, -0.109522, -0.1275, -0.059184, -0.0980703, -0.102617, 0.0133898, -0.104578, 0.792576]\n"
     ]
    }
   ],
   "source": [
    "# Note that each gradient has the same size and shape as the corresponding parameter:\n",
    "@show ∇b = grad(J,model.b);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Checking the gradient using numerical approximation\n",
    "\n",
    "What does ∇b represent?\n",
    "\n",
    "∇b[10] = 0.79 means if I increase b[10] by ϵ, loss will increase by 0.79ϵ"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "value(model.b) = [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "19.10423456298375"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Loss for the first minibatch with the original parameters\n",
    "@show value(model.b)\n",
    "model(x,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.1"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# To numerically check the gradient let's increase the last entry of b by +0.1.\n",
    "model.b[10] = 0.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "value(model.b) = [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.1]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "19.183620170313954"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We see that the loss moves by ≈ +0.79*0.1 as expected.\n",
    "@show value(model.b)\n",
    "model(x,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Reset the change.\n",
    "model.b[10] = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Checking the gradient using manual implementation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# Without AutoGrad we would have to define the gradients manually:\n",
    "function nllgrad(model,x,y)\n",
    "    scores = model(x)\n",
    "    expscores = exp.(scores)\n",
    "    probabilities = expscores ./ sum(expscores, dims=1)\n",
    "    for i in 1:length(y); probabilities[y[i],i] -= 1; end\n",
    "    dJds = probabilities / length(y)\n",
    "    dJdw = dJds * x'\n",
    "    dJdb = vec(sum(dJds,dims=2))\n",
    "    dJdw,dJdb\n",
    "end;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([0.0 0.0 … 0.0 0.0; 0.0 0.0 … 0.0 0.0; … ; 0.0 0.0 … 0.0 0.0; 0.0 0.0 … 0.0 0.0], [-0.139954, -0.064541, -0.109522, -0.1275, -0.059184, -0.0980703, -0.102617, 0.0133898, -0.104578, 0.792576])"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "∇w2,∇b2 = nllgrad(model,x,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "true"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "∇w2 ≈ ∇w"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "true"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "∇b2 ≈ ∇b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Training with Stochastic Gradient Descent (SGD)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "sgdupdate! (generic function with 1 method)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Here is a single SGD update:\n",
    "function sgdupdate!(func, args; lr=0.1)\n",
    "    fval = @diff func(args...)\n",
    "    for param in params(fval)\n",
    "        ∇param = grad(fval, param)\n",
    "        param .-= lr * ∇param\n",
    "    end\n",
    "    return value(fval)\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "sgd (generic function with 1 method)"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We define SGD for a dataset as an iterator so that:\n",
    "# 1. We can monitor and report the training loss\n",
    "# 2. We can take snapshots of the model during training\n",
    "# 3. We can pause/terminate training when necessary\n",
    "sgd(func, data; lr=0.1) = \n",
    "    (sgdupdate!(func, args; lr=lr) for args in data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "model(dtst) = 2.313187215962106\n",
      "3.03e-01  100.00%┣███████████████████┫ 6000/6000 [00:10/00:10, 598.92i/s]\n",
      "model(dtst) = 0.2806535683801265\n"
     ]
    }
   ],
   "source": [
    "# Let's train a model for 10 epochs to compare training speed on cpu vs gpu.\n",
    "# progress!(itr) displays a progress bar when wrapped around an iterator like this:\n",
    "# 2.94e-01  100.00%┣████████████████████┫ 6000/6000 [00:10/00:10, 592.96/s] 2.31->0.28\n",
    "model = Linear(784,10)\n",
    "@show model(dtst)\n",
    "progress!(sgd(model, repeat(dtrn,10)))\n",
    "@show model(dtst);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Using the GPU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "model(dtst) = 2.3035536f0\n",
      "3.03e-01  100.00%┣██████████████████┫ 6000/6000 [00:04/00:04, 1654.96i/s]\n",
      "model(dtst) = 0.28049186f0\n"
     ]
    }
   ],
   "source": [
    "# The training would go a lot faster on a GPU:\n",
    "# 2.94e-01  100.00%┣███████████████████┫ 6000/6000 [00:02/00:02, 2653.45/s]  2.31->0.28\n",
    "# To work on a GPU, all we have to do is convert Arrays to KnetArrays:\n",
    "if gpu() >= 0  # gpu() returns a device id >= 0 if there is a GPU, -1 otherwise\n",
    "    atype = KnetArray{Float32}  # KnetArrays are stored and operated in the GPU\n",
    "    dtrn,dtst = mnistdata(xsize=(784,:),xtype=atype)\n",
    "    Linear(i::Int,o::Int,scale=0.01) = \n",
    "        Linear(Param(atype(scale * randn(o,i))), \n",
    "               Param(atype(zeros(o))))\n",
    "\n",
    "    model = Linear(784,10)\n",
    "    @show model(dtst)\n",
    "    progress!(sgd(model,repeat(dtrn,10)))\n",
    "    @show model(dtst)\n",
    "end;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "## Recording progress"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "trainresults (generic function with 1 method)"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "function trainresults(file, model)\n",
    "    if (print(\"Train from scratch? (~77s) \"); readline()[1]=='y')\n",
    "        # This will take every nth element of an iterator:\n",
    "        takeevery(n,itr) = (x for (i,x) in enumerate(itr) if i % n == 1)\n",
    "        # We will use it to snapshot model and results every epoch (i.e. 600 iterations)\n",
    "        lin = ((deepcopy(model),model(dtrn),model(dtst),zeroone(model,dtrn),zeroone(model,dtst))\n",
    "        # (progress displays a bar like progress! but returns an iterator, progress! returns nothing)\n",
    "               for x in takeevery(length(dtrn), progress(sgd(model,repeat(dtrn,100)))))\n",
    "        # Save it as a 5x100 array\n",
    "        lin = reshape(collect(flatten(lin)),(5,:))\n",
    "        # Knet.save and Knet.load can be used to store models in files\n",
    "        Knet.save(file,\"results\",lin)\n",
    "    else\n",
    "        isfile(file) || download(\"http://people.csail.mit.edu/deniz/models/tutorial/$file\", file)\n",
    "        lin = Knet.load(file,\"results\")    \n",
    "    end\n",
    "    return lin\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train from scratch? (~77s) stdin> n\n"
     ]
    }
   ],
   "source": [
    "# 2.43e-01  100.00%┣████████████████▉┫ 60000/60000 [00:44/00:44, 1349.13/s]\n",
    "lin = trainresults(\"lin113.jld2\",Linear(784,10));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Linear model shows underfitting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "using Plots; default(fmt = :png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlgAAAGQCAIAAAD9V4nPAAAABmJLR0QA/wD/AP+gvaeTAAAgAElEQVR4nO3deXxV1b338d/aZx6SnMyQAJF5dCpaFRVEWyxqQL2P1Vst1nqvr6rU2mvVvhzooKjgba/D01e92kZbWi+tPoJSWm0d0Cve9lqrosigogwhQAiZzsmZ93r+2CHGkCAIOSdkfd5/8Dpn7X3O/p3NTr5Ze+19ltJaCwAAprLyXQAAAPlEEAIAjEYQAgCMRhACAIxGEAIAjEYQAgCMRhACAIyWoyBsbm6ura0tKSmZM2dOc3Nzr+u8++67oVAoN/UAAODIURAuWrSopqamoaFhxIgRixcv3neF1tbWb3zjGx0dHbmpBwAAh8rNN8uMHz/+6aefnjBhwvr16+fOnbthw4buS7XWF1544aWXXnrRRRfxTTcAgFzKURCGw+HGxsZAIBCPxysrK9va2rovveeee3bv3v3v//7vSvVeT1NT09VXX51IJHq0+3y+xx9/PJvNKqUsq/fe7dZtOwKhcFlx+HB9Fjj2v9vRTzKZjMvlUkrluxCzpNNpj8eT7yqMc1C73eVyfe5fR+7P97KDpbV2fnS11tlstvuil1566dlnn/3LX/6yn5c/+eSTbW1tV155ZY92l8uVzWbb2tpcLldf44sbn3gsNfHUWV+edmifAD21tLT4/X6/35/vQsyyZ8+ewsJCfinn2O7du8vKyvizL8d27do1ZMiQA1z5UP53chSEVVVVW7duHTt2bH19fXV1dfdFL7zwwssvv+z1ep2nSqn//u//Pu2003q8Q01NzUUXXdTrm6dSKZfL1ddvZKWU2+3m9/Vh5/P5CMLcc3Y7QZhjzm4nCHPM2e052FCO/l9ra2vr6uq01nV1dXPnznUaV61aJSJ33nmn3ktEtNb7puCh0Eppm3FHAEDvchSECxYsWLNmzfDhw9euXXvbbbc5jTNnzszBppUoLQQhAKB3OTo1GolEVq5c2aNx3+ti+uXKHSXClagAgD6YcMqb6+sAAH0yIQg5MQoA6JMBQaiUcLEMAKAPgz8ItXO5DAAAvRn8QSiKc6MAgD6ZEIScGgUA9MmAIKRDCADomwlByLlRAECfDAhCpbihHgDQl8EfhB4lsUy+iwAADFSDPwjDXtWYsPNdBQBggBr8QVjgUY3xfBcBABiojAjC3QnGCAEAvRv8QRjySiyt4wwTAgB6M/iDUIkq9cnHUTqFAIBe5Gg+wnxSqsKvN7XLxEi+KwEAkR/84AfPPPNMvqsYuMaPH7906dJcbtGAIBQp98uHbZqJCQEMBJs3b77gggtqa2vzXchAtHHjxrvuuivHGzUgCJUq98lL7ZwaBTBQjBgx4vjjj893FQORZeVhwG7wjxGKSHlANrXluwgAwIBkQBAqVebVH7bRIwQA9GLwB6ESVRaQj6J83ygAoBeDPwhFxGtJkUcaOohCAEBPBgShEtEyqlB9yDAhAGAfBgShKBE9ukBt4sJRAMA+DAhCpUTrUYWyietlAAD7MCYIC9SH7fmuBAAw8Az+IFS+gB2PjS5U9AgB4NApNdi+pWvwB6G7rCqzu35UgfqQMUIA6MOXvvSlfJeQN4M/CD0V1Zld9UOCEktLezrf1QDAgPTCCy/ku4S8GfxB6C4flmncpkRGFqiP6BQCwD7OP/98ETnuuONERCm1ZMmSIUOGOI+XLl167LHHlpaW3nfffd1fsmfPnnnz5g0dOrSqquryyy/fs2eP015XVzd06NCysrIHHnigr5aBZvB/6bYVLhJRdrR1VGF4U7s+pmSwnd0GcKTbFZerXs3mZlsVAXn4NFePxuXLlyul3nrrLefp3/72t64O4pYtW956662XXnrpvPPOu/7667tecv3113u93k2bNonIddddd8MNNzz66KMicsMNN7zyyis+n+/aa6+97rrrem0ZaAZ/EIqIu7w607htdMEE7qkHMACFPfKNcTn6Gz3o/uwN/eAHPygvL3ceX3311UqpM888Mx6Pd1/nj3/843vvvRcIBETkzjvvPOaYY5z26dOn33LLLfPmzfvTn/7UV8tAY0YQVlSnG7ePLJi4vpVTowAGnKBbzq8ZQANVXSkoIgUFBX2t1nX5qFIqm+3s0S5fvvzPf/7zY4899vDDD//lL3/ptWWgGUC7vv+4y6ozjfXcQQEA+5FOH8T1hLNnz7711lsTiUQ8Hr/11lvPOeccp33kyJEjR45csGDBG2+80VfLQGNGEFZUZ3bVjyoQ7qkHgF6dc845o0ePPvD177vvvng8ftRRR40aNSqVSnVdSnPTTTedcsopZ5xxxr333ttXy0BjxKlRT/mw9sZtIwvU1qjOanFxuQwAfNrKlSudB7rbnHW9PnYelJaWLlmyZN/3ueaaa6655pr9tww0xvQId2/3Wbrcr7bFODsKAPiEEUGovH4rEM62NI4qFC4cBQB0Z0QQSucdFPVMxgQA6MGYIKwYlt5VP4oLRwEAn2ZMEJZVZRq5cBQA0JMxQVgxLNO4jVsJAeBzG3wTMDlMCUJPuXMrIWOEANCL/UzDNOhnaDIlCF1lQ7MtjaWejK2lOZnvagBggNnPNEyDfoYmU4JQudyuSHl2d8NIOoUA8Gndp2HqMWtS90WOwTQBk8OIb5ZxuCuq0431owqrNrXrqWWD80w3gCOR3dHe9uxvcrMtK1hQ+JXLejR2n4apx6xJPWZoksE1AZPDpCAsH5ZprD+m5Iv/u0tfNDLf1QDAXsrldpdV5Whb/uD+V/jMWZMG0wRMDqOCsDq9fdO5R6tLXrTvPSnf1QDAXsoXCE+fm+8qOh3IrEmDZgImhyljhCLiqajONNZ/oUwlsrKRiQkB4NOcaZh6nTWp+wxNg2kCJodBQegur87s2qZEzhuhntlCEALAJ7qmYdp31qQeMzQNpgmYHAadGnVFyu14VCfjtSP8i97Ofu9og/4IAID965qGad9Zk3rM0DSYJmBymBQGSrnLqjKN28+qUmv26N2JfNcDABgATApCEXf5sHTjNp9LzhhqPbvNznc5AID8MywIK6ozu+pFpLZGrWCYEABgXBCWV2d214vIecOtP2+zU/QJAcB4ZgWhp7zK6RFWBGRCRL3cQKcQAExnVhC6y4dlGrc5j2tHWCu20CUEANOZFYRWuEhE2dFWEZlTo57ZTI8QAExnVhCKM0zYuE1EphQrS8k7e8hCADCaeUFYUZ3e2XV2lK+YAQDTGfTNMg7f2OMS77wWOvlsEakdYd32RvbW44z7awBAfj333HNNTU35rmIgamhoyP1GjQvC4LGntT71kB1rs0KFM4aq91v1jrgMCeS7LADGuOeee+69997t27fnu5ABat68eTneonFBqHwB38Sp8bdfDU07x2PJl6utpz6yr5lEpxBAjgwZMuQnP/lJvqvAJ0wMgODUMzveeMl5/J0p1uI13FkPAOYyMQj9E09I79ic2bNTRE6pUOOL5Nfvk4QAYCgTg1C53IFjT4u/+bLz9EdTXXe8SacQAAxlYhCKSPCEMztef955fHKFmhiRxzaShABgIkOD0Ddysk4l09s/cp7eeYJr4Vt2MpvfogAAeWBoEIpSgS+c0fHGi86zE8rU5GJ5lE4hAJjH1CAUCZ5wVsffXxTd+c0yd0x13f02nUIAMI65QegZMsIKFyU/fNd5OrVMHV0sv9xApxAAzGJuEErnDYUvdj398VTXwrfseCaPFQEAcs3wIJwZX7NaZ9LO0y+UqRPK1X1r6RQCgEGMDkJXUamnamTi3b92tfxsmnX/u9n/2cWUFABgCqODUEQKzvpq6x/qdDrlPB0WUr843f31VdnWVH7rAgDkiOlB6J8w1TtifNtzv+1qOW+EOnuY+tf/5vpRADCC6UEoIpF/uqbjf/+crv+wq+WnJ7neb9NcQQoAJiAIxQoVFs6e1/y7B7ruKfS55Pdnum79e3ZdC4OFADDI5S4Im5uba2trS0pK5syZ09zc3GPps88+O2nSpEgkMmnSpD//+c85q8oROvkryuePvrayq2VskVp4guurL2S5mwIABrfcBeGiRYtqamoaGhpGjBixePHi7ots27700ksffPDBPXv2/PjHP77iiityVlUnpYov+nbbn36TbW3qartyvHVMiZr3cjbDKVIAGLxyF4TLli2bP3++z+ebP3/+U0891X1RJpNZsmTJmWeeGYvFfD5fJBLJWVVd3BXDwqee07Lsoe6Nj85wJbPyzy+RhQAwaLlztqX6+vqamhoRcfqF3Rd5vd5zzjknGo0WFhYqpV599dV9X7569eqvf/3rPRp9Pt9PfvKT9vZ2l8uVyRzqSUx94uzkz29qWv2se8opXY2/PFG+/prvq39J/+KkpEsd4hYGlba2tlQqlUwm812IWdra2kTE4/HkuxCztLW1eb1ey+Kiipxqa2sLBoMHuHIwGPzcPxe5C0KttVLKeZDN9nJzQjgcjkaj999//3e+853XX3+9x9Ly8vJp06b1aHS73R6Px+PxWJZ1GH41eDyuf/5e+2N3uEMFnnHHd7aJ/PZ0+/+87LruDf9DJ2ctsnAvz175LsQs7Pa86Po9k+9CzHJQh/qh/O/kLgirqqq2bt06duzY+vr66urq7os+/vjjn/3sZ/fee28oFLryyivvuuuufV8+bty4q6++utd3zmQyLpfrwP9w2J8xk33/+sPdj/yw9PLv+8Ye57QFRVbOlvOey1z3hvsXp7vIQkcymQwEAn6/P9+FmKWjo+NQ/vLF5xMIBILBIEGYY85uz8GGcvf/WltbW1dXp7Wuq6ubO3eu07hq1SoRqaqq+uUvf/nyyy9rrX/3u98df/zxOatqX96aCWVXLmj61T3JD9/pagy6ZcXZ7g/a9KWrsh1cRwoAg0jugnDBggVr1qwZPnz42rVrb7vtNqdx5syZIuL1epctW/Zv//ZvpaWlS5cufeSRR3JWVa+8IyeVzru56dGFqS0buxpDbvnLbHfILac8k9nUzv2FADBI5O7UaCQSWblyZY9Gvfce9hkzZrzxxhs5K+Yz+cYdX/LP32165Ael//pD74jxnY0u+cXprofX26c+k/nVGe5Z1ZwkBYAjHqe8++SffFLxJdfvfvgH7aue6vrSGRG5aoL1xFnub76SXfS2TccQAI50BOH++CefVHnDA4k1qxt/fku2bU9X+2lD1F/nuJ762K59LrMtRhoCwBGMIPwMruKK8vmLfaMm7/r3+Yl1f+9qHxZSq2vdpw+xvrAsc/+7Nn1DADhCEYQHwHIVfuWykstual76Hy1P/l872uI0uy25+VjrtTnuZZvtGSszG1oJQwA48hCEB8o37rjKmx8St3fH3Ve1/fm/dCrhtI8pVC+c4/6no6zTVmTueNOOcXMFABxRCMKDYAULIudfVfFvD2R2bN6x8MrYX58VOysiLiXXT7Fen+te36LHP5F5eL3Nd5MCwJGCIDxo7tIhJfO+X/ovP+z4x6qGO65of/53dqxNRI4qUL+d6XruK65nNttT/l/miY8YNwSAIwBB+Dl5h48tv+aesit/kNmzc8fCbzY9tjD18ToRmVys/nC2+/5TXAvftE9cnnniIztLHgLAAJa7G+oHJc+w0cVfva7o3G/E/vps06/vtoIFweNnBI6fcfawylnD3H/YYi962775f+3vTLaummAF2NkAMPDwu/kwsEKFBWd9teDMi5Ifvht/8+VdP/2Ou3RI8Pjps48+pbZ26Ks79OI19qI16asmWP863qoO8X00ADCAEISHj1K+MUf7xhwd+adrEu+/FX/zlfYXn1S+wJSJJ/x+wtSPjj/6/77vPeapzOlDrG9NtGZVK2axAICBgCDsB5blH/8F//gviNbp7R8l1r/R/tL/K9xy9w+rR/9o5JTVeuLdfx1/TTZ8+Vjr0jFqTCF5CAD5RBD2J6U81aM81aMKzrpIJ+PJj9elNq2d9v6yE7ZsSBdWvrdz/H0vj26tGHPylFEXjfNXBPJdLQAYiSDMEeULdHYTRcTOprZ9WLZl40lb3t/z8XPuf2xd4x3WVDKqZHjN5ImjKmtGuCLl+a4XAExBEOaD5fKOGOcdMS4sUiqi06nSbR+/vXZT/cebX3vyjQnJLSFJuSqGhYcO81QMc5dXu8ur3eVVystc8ABw+BGE+ac83sKR404fOU5EUra8tF0/u6Fl44dbRzfXT2/fPnHjqpL2ertpu+UPuUor3aVD3SVDXKWVrki5q7jcXVypvL58fwIAOIIRhAOL15Kzh6mzhxXLWcXv7Dn6T9v0z7bab+zWJx+rzi1umeHbMSazy97TkPpoXbbllUxLY3bPTuX1u4pKXZFyV2GJK1LmKip1FZZYBcWuwhKrIKJc/BcDwP7wW3LgOrpEHV2ibjrGakvLyw32C9uLf/lRZFts/PQh1oyj1WmV6vhS5bbEjrZmW5uyrbuzrU3Z1qbU5g3Z9ma7vTnb2mRHW5U/aIUjrnCRFY64CiJWuMgKF1nBAldBxAoVWaFCK1igPN58f1YAyBuC8AhQ6JHaEVbtCBGRXXF5cbv9yg5dt8HeEtUnVajThhScVll44thRBZ5eXmtHW7PRFjvamm1vtttb7FhbZsfmbLTVjrXZ0Va7o93uaBcRK1hgBQusUIEVKLCCYRUIW4GQFQhbgZDyh6xAyPKHVCBoBQosf0AsV253AAD0I4LwCFMRkEtGW5eMFhHZk5TVO+1Xd+gf/sN+s0mPLFAnlatTKtUXy9XEiHIpERGnC7j/99SppJOIdke73RG14+12PKbj0UxjvZ2I6XjMTsTseIdOdNjxdjvRoVxu5QuK158MhFyBkPIFLJ9f+UOWP6g8PuX1WYGw8vqVx2MFwsrjFbfXCoSV26O8fuXzc7YWwIDCr6QjWInvk55ixpa39+i/7tKrtut719jbYvrYEjW1TJ1Qro4vVROKlLvv71dXXp/L63NFyg5wuzqVtJMdLTsbfEp77IxOxe1kQic67ESHTiXsjvZM0w6dSuh0yo7HdDolmZTdEdWZlE4n7URctG0FQsrtUR6/8vqU26MCIWW5lD+oLEv5Q8pyKV9AuVzKFxDLZTn/+oOixAqERUT5Q6JU52tdbuUNiIjlD4rFl8gDOGgE4SDhtmRqmZpapq6dJCLSnpZ/7NZv7NZ/2qoXvmlvjemJEXV8qTq2VB1boqYUq+JDuNTUCU4rI+5AwOc/+Js6tLbjMScXdTKhsxkdj2k7qxMdnf9mMzqV0Nms3RGVbCaTSkg2ayc7RIsdj4qIjsdEtJ1KSCajsxmdiouIHY+J1srtcS6jVf6gUi5xuSxfQETE7VEen4hYXp+4PSJi+YLicomIE7TO51Juj4gol7vrZhUrEBalnHew9l6gq7wB57UiYgVCnSs4K4vatx3AQEYQDk4FHpkxVM0Y2vmLOJaRd/bot5r0W0368Q/stc26yKumlMgxJWpysZoUURMiKpSzY0EpKxjup/fW6ZROp0TETsRE25LN2sm4iEhmb3sqKZm0iNjJDslmRcROxp0JlnUqYXckRcRJYucNnTFUEZFM2k4lO7eSijuvFRE7HpW9M23Z8b0r7w3mzk+8N547n3r9yvXJiK4KhFS3yFRev3Q7e6zc7u63kKaSyVZ/wBUI9fjgTl/5U5Sy9lmt9zWd1f1B1cfor9NZ73WRiFK9beVTWzzgznr3vyQObH3+2sBhQBAaIeSWkyvUyRWdvzK0yMft+t1m/W6zPLtV/8c79oZWXRlQEyMyMaLGF6nxETUxosqPwDv4lcfrXATbf1n7OehMWu8NURHRqYTOpj95Go9p/cmslTqZEDvT7bWfpLKIZNvbfT6vlUn12ITTV/70VrXdsU+jSKZpR+9FJjq0ne17kd3rIhGt47E+Fu2tLdEhfb7802vGoyIHMYFn9782PrcDyWnbtnfk9sS7cnudcxiHyDn1cujvk3t2okOddr5ceFUOtkUQmkiJjCxQIwuUM74oIraWj9r1ey16fYv8rVH/6n17fYvWImOL1NhCNa5IjS2UMUVqTKGKcKvFwVNuj3PStdMhhHRs925/UZHH09slwvhcDiSnGxsbS0tLrRxm4eEKsMMVqLln+YMNO3fmZlsEIURELCWjC9Xowk+iUUSakrKhRW9s1e+36ac+lg/a7A/atNeS0YVqTKGq9nrHRtS4Ej2qUKqDzCqFI5XlD372Sv6YFQznMgiRSwQh+lTqk2mValrlpyJuV1w+bNcftul3d8rLO9WvP8p+1C67E3pEWB0VlqMKVE1Y1YTlqLA6qkCGBjvv4gCAAYsgxMGpCEhFQJ1SoZpLU4GAy+/3ikgyKx+164+jsjmqN7frP22Vj6P25qjsTughATUiLDVhNTwkw0JqeFhGhFV1UJUdgQOQAAYlghCHgc8lEyJqQkR6XPKXtmVbTG+Nyeao3hKVd5v1n7bpLVHZFtPxrAwLqaqgDA+p6pBUBVV1UIYG1fCwVAaUl1NQAHKFIEQ/8ljOVTmy7zXx8YxsjentHbI1putj8kGbfrlBGjrsbTHZEdclPqkIqGEhqQyo6qBUBtTQoAwJqMqAVIVyeKcHAAPwGwX5EXDLuCI1rkj2zUgtsisuO+N6W0x2xfW2mLzfpl/ZITvi9s64bI9pERkSVEMCUu5XVSEp90u5Xw0JSEVAlfulMnBIXxcAwDQEIQYcJVIZkMqAOqZEer29OpaRhg69My6NCV0fk8aEXtusn6+XxoTdmJAdHTqelXK/KvNLZUAqAqrML2U+VRGQioCU+lSZv/NfrnQFIAQhjkQht4wpVGMKpa9vIUlmZXdCNyZkR1x2J/TuhOxO6P9tlN0J2Z2wm5KyO6GbElLql1KfKvV3paOU+Z2nUuJTpX4p8UmJT/mZbAMY1AhCDEI+l1SHVHXnN3/1Hpa2lqakNCX07oQ0JXVTUpoS0pjQ77dJU0KakvaepOxJ6j1JcSkp8akSX2culvikuNuDYq8q7vYAwBGHIIShLNU5uCgi+/9+y1hG9iR1U0KaU7InofckxcnITe3SnJTmpL0nKc0paUnqlpQU+yTiVcU+iXgl4lURr0R83R9IkVdFvFK09wGAvCMIgc8QckvIrYbvt3/p0CItSWlO6eaktOyNxpaUtKT0xtbOB60paUlJa0paU7o11RmNhU40eqTQq4q8UuSViFcVeqTQK4UeKfCqQo8U+6TAo9I2A5vAYUYQAoeNEin2SbFPSUFXw2doTkprSrelnWiUtrRuTUlbSppTektU2tLSlpL2tN2WlpaktKV1W6pQRAo86SKvKvJKgUcKPBL2qKK9kRl2S9gjxT4Ju1XYIyG3RHxS4FEhtwT5cQd6w08GkE+dwfmJz8jO3bt3+8NFCfG0pXVLUtrT0p6WaLozStvTeltMohlpSUp72o5mJJbuzNdYRhIZifgk7FYhj4TcUuyTkFsF3VLgkSKvBN0SdKuIV0JuCbil0KMKPBJwS9gjhR4VcAu3b2Kw4tAGjjA+l4Q9Uubv6nfKAc7hZ+vOsIxlJJaRlqREM7ojI9G0tKYkltG7E/rDNollpCMj7Wm7LSXxrBOlOp6VjowUeSXgkqBbRXwScEnALRGv8rsk6JYir/hcnd1Qn8tZU/ldEvGJs0KBR/lcUsi0GRh4CELAFJY66A5oDy0piWckntXNSUlkJZ6RlpSOZyWekdaUJLPSnNJbYpLMSltKOjJ20pbmpCSz0pGRtrROZqU9LUG3k5TKa4nT6fTv87TQozxWZ7gG3RJyK68lRV5xW1LkFa8lIbdy1gQOHUEI4EBFvBLxishBd0a768hIMistKZ2yJZr+5Gnalva9T1tTOm3LB22dcRvN2GlbWpKS0dKWkpQtsYx21izwiNuSiFe5lBR5xWNJ2CN+lwRcykncsEc8lhR6lEtJxNc5jmuJFHmVy5JCj7iUFHrFraTAo9yWFNBnNQ9BCCCngm4Junt0TOVzBKqjLS1ZW5pTOmtLW1pStsTSkshKPKtjaUnZ0p6WjC1taZ3V8lG72CItSclqaUvbmb1L29OS0dKe1mlbomlx4rB7NNqZwpJg1m3ZhR5lKSnyihKJ+EREir1KRIq8YqnOVHb6r36XBNydwaxEIt1Ww0BDEAI4gjmDjvvEqnzuZBURJw7TtkQz2onJXbtb3eFiW1RbujNxnQFXEWlOaRH5qF20dMZqLGOn7M6+rPNWtkhrSotIS1K0dHZSnW6r0ysVkYhXlJKwW3mszhPCTuKKSIFHuVVno4g439vgxK0TtF0vd8Zlu16IA0QQAsCneKzOsKnYm6Y7s5nycrEsdSj52sVJWafb6gStiDQnRUTa0zqjO0O0K2ud7qwz1OrcqyrSebrYCVoRaUmJ1hLP6kT2kxeG3OJ1dZ4HFpGQR7zWJzEZdCufJbI3WbuC1jmN3HWW2HmTrk6t19V5/bDTLiJFXmWJHNFnlQlCAMip8CeBcTg7svuKZSSVdc4DaxFxzhV3xWRHRidtkb0Z7AStiLSmtC2fRKwT21qkJWV3X815cxFpSWkt0pXoztCsSGfvVvaeEHbGYkU6r3USka7Lnbpe4nRzRaTQKy4lR4l76NDDuD/6RBACwOAU2nv3Z5m/13ztl+FK5yIm2du7lb0nhLNa2lIiIklbOjJaROIZSWRFumVqNKNjGRGRzVHJagkV5GiGboIQAHDYBN1dY5l7g7Zg37UOKIMbGlKHraz9ylHeAgAwMBGEAACjEYQAAKMRhAAAoxGEAACjEYQAAKMRhAAAoxGEAACjEYQAAKMRhAAAoxGEAACjEYQAAKMRhAAAoxGEAACjEYQAAKMRhAAAoxGEAACjEYQAAKP1EoTr1q078cQTlyxZIiI33nhjOByeOXPm1q1bc14bAAD9rpcg/Pa3v11ZWTl79uwtW7Y89NBDTz75pNfr/e53v5v74gAA6G/ufZv+9re//eIXvygrK1u8ePHs2bO/8pWvtLa2XnvttbkvDgCA/tZLj9DtdiulRGT16tUzZswQkUAgkEqlcl0aAAD9r5cgPOmkk1asWLFu3boXX3zxoosuSqfTS9TqzyoAABDiSURBVJcuPfbYY3NfHAAA/a2XIFy8ePHzzz8/adKkiy++uKKi4tprr33xxRfvvffe3BcHAEB/62WM8Jhjjtm6devOnTurqqpEZNGiRT//+c9dLlfOawMAoN/1EoQi4na7q6urncfFxcU5rAcAgJziPkIAgNG4jxAAYDTuIwQAGI37CAEARuM+QgCA0XJ3H2Fzc3NtbW1JScmcOXOam5t7LH366aenTJkSiUSmT5++cePGQ9wWAAAHqJcgdO4j3LZt2yOPPCIiixYtqq+vP/nkkw9xS4sWLaqpqWloaBgxYsTixYu7L9qyZctll132yCOPNDQ0zJkz54orrjjEbQEAcIB6n4/QuY/QGSksLi4+LHfTL1u2bP78+T6fb/78+U899VT3RZs2bbrkkktOOeWUQCBw+eWXb9iw4dA3BwDAgej9hvoVK1YsXrx43bp1tm1Pnjz5+9///rnnnnuIW6qvr6+pqRERp1/YfdEZZ5xxxhlniEg2m12wYMHFF1+878v//ve/73sLh9frve2226LRqMvlsm37ECvEQYlGo5lMJp1O57sQs0SjUcuyPB5PvgsxSzQa9fv9lsVM5jkVjUbb29sPcOVAIOB2955on6mXlz3xxBNf+9rXbrzxxrvvvlsptXLlyrlz5/7+97+/8MILP982HFprp4uptc5ms/uu8Pzzz990002zZs268847913q8/lKSkp6Vu92q24OpTwcLHZ7XrDb84Ldnhc52+e9BOFdd9110003LVy40Hl66qmn2ra9cOHCQwzCqqqqrVu3jh07tr6+vuv72xxa61tuuWX16tVLly4dN25cry8/+uijb7/99l4X2bbtcrlCodChlIeDlU6nA4GA3+/PdyFmSSQS4XCYHmGOxWKxcDhMjzDHQqFQOBzOwYZ6+X/duHHjaaed1r1lxowZhz5uV1tbW1dXp7Wuq6ubO3eu07hq1SoRee2115YtW/bMM89UVVVFo9FoNHqI2wIA4AD1EoQ1NTVr167t3vLuu+86w3uHYsGCBWvWrBk+fPjatWtvu+02p3HmzJkismrVqg0bNhQXFxfsdYjbAgDgAPVyavSqq666/fbbKysrnQtkVq5c+eMf//iOO+44xC1FIpGVK1f2aNRai8itt9566623HuL7AwDwOfQShNddd10mk/nud787b948ESktLV2wYMF1112X89oAAOh3vQShZVnf+973brjhhsbGRhEpLy/nWikAwGDV510XSqmKiopclgIAQO4d0NXAy5cvp1MIABiUuC0GAGA0ghAAYDSCEABgNIIQAGC0zqtG169fv5+Vtm3blpNiAADItc4gnDhxYn7rAAAgLzqD0PmqMwAATMMYIQDAaAQhAMBoBCEAwGgEIQDAaAQhAMBoBCEAwGgEIQDAaAQhAMBoBCEAwGgEIQDAaAQhAMBoBCEAwGgEIQDAaAQhAMBoBCEAwGgEIQDAaAQhAMBoBCEAwGgEIQDAaAQhAMBoBCEAwGgEIQDAaAQhAMBoBCEAwGgEIQDAaAQhAMBoBCEAwGgEIQDAaAQhAMBoBCEAwGgEIQDAaAQhAMBoBCEAwGgEIQDAaAQhAMBoBCEAwGgEIQDAaAQhAMBoBCEAwGgEIQDAaAQhAMBoBCEAwGgEIQDAaAQhAMBoBCEAwGgEIQDAaAQhAMBoBCEAwGgEIQDAaAQhAMBoBCEAwGgEIQDAaAQhAMBoBCEAwGgEIQDAaAQhAMBoBCEAwGgEIQDAaAQhAMBoBCEAwGgEIQDAaAQhAMBoBCEAwGgEIQDAaAQhAMBoBCEAwGgEIQDAaAQhAMBouQ7C5ubm2trakpKSOXPmNDc377tCNpudMGFCjqsCABgr10G4aNGimpqahoaGESNGLF68uMfS+++/f9q0aRs2bMhxVQAAY7lzvL1ly5Y9/fTTPp9v/vz5c+fOvfvuu7svPeaYY0aPHl1bW7vvC999992FCxf2aHS73ddee20sFnO5XFrrfqwb+4jFYtlsNpPJ5LsQszhHu8fjyXchZonFYoFAwLIYS8qpWCwWjUYPcGW/3+92f85Ey3UQ1tfX19TUiIjTL+yxdObMmX29MB6P79q1q0ej1+vV3Rz2arEf7Pa8YLfnBbs9L3K2z3MdhFprpZTzIJvNHvgLTzzxxPvvv7+v93S5XKFQ6PCUiAOTyWQCgYDf7893IWZJJpMFBQX0CHOso6OjoKCAHmGORaPRgoKCHGwo1/+vVVVVW7duFZH6+vrq6uocbx0AgB5yHYS1tbV1dXVa67q6urlz5zqNq1atynEZAAA4ch2ECxYsWLNmzfDhw9euXXvbbbc5jfsZGgQAoF/leowwEomsXLmyR2OP4VBGpAEAOcPYLwDAaAQhAMBoBCEAwGgEIQDAaAQhAMBoBCEAwGgEIQDAaAQhAMBoBCEAwGgEIQDAaAQhAMBoBCEAwGgEIQDAaAQhAMBoBCEAwGgEIQDAaAQhAMBoBCEAwGgEIQDAaAQhAMBoBCEAwGgEIQDAaAQhAMBoBCEAwGgEIQDAaAQhAMBoBCEAwGgEIQDAaAQhAMBoBCEAwGgEIQDAaAQhAMBoBCEAwGgEIQDAaAQhAMBoBCEAwGgEIQDAaAQhAMBoBCEAwGgEIQDAaAQhAMBoBCEAwGgEIQDAaAQhAMBoBCEAwGgEIQDAaAQhAMBoBCEAwGgEIQDAaAQhAMBoBCEAwGgEIQDAaAQhAMBoBCEAwGgEIQDAaAQhAMBoBCEAwGgEIQDAaAQhAMBoBCEAwGgEIQDAaAQhAMBoBCEAwGgEIQDAaAQhAMBoBCEAwGgEIQDAaAQhAMBoBCEAwGgEIQDAaAQhAMBoBCEAwGgEIQDAaAQhAMBoBCEAwGiDIQjb2tpisVi+qzBOc3NzIpHIdxXGaWpqSqfT+a7COLt27bJtO99VGGfnzp252dBgCMK77rrrsccey3cVxrn++uufffbZfFdhnEsuueTtt9/OdxXGOeuss3bt2pXvKsySTqdPPPHE3GxrMAQhAACfW+6CsLm5uba2tqSkZM6cOc3NzQe1FACAfpK7IFy0aFFNTU1DQ8OIESMWL158UEsBAOgn7pxtadmyZU8//bTP55s/f/7cuXPvvvvuA18qIqtXr7788st7NLpcrpNOOum9997bsWPHf/7nf/bvB8Cnbd68+fnnn29tbc13IWZpampavnz5mjVr8l2IWeLx+G9+85uioqJ8F2KQbDZr2/aB/2KfNWvWyJEjP9+2lNb6873yYIXD4cbGxkAgEI/HKysr29raDnxpNBqdNm1aPB7v8Z6WZZ1yyinJZFIp5fV6+/0zoJtEIuFyuTweT74LMUtHR4fP53O5XPkuxCyxWCwYDCql8l2IWaLRaDgcPsCVv/3tb0+dOvXzbSh3PUKttXMYaa2z2exBLQ2Hw/wJDADoD7kbI6yqqtq6dauI1NfXV1dXH9RSAAD6Se6CsLa2tq6uTmtdV1c3d+5cp3HVqlX7WQoAQH/L3RhhS0vLpZde+vbbb3/hC19YsmSJM+ysVGcBvS4FAKC/5S4IAQAYgI74b5bhTvycefrpp6dMmRKJRKZPn75x40an8dRTT1V7fetb38pvhYNSr3uYw76/qX0IR3u/yWazEyZM6N7S6xHef4f9ER+E3ImfG1u2bLnsssseeeSRhoaGOXPmXHHFFSKitV6/fv22bdva29vb29vvu+++fJc52PS1hzns+1t7N7fffvvNN9/M0d5P7r///mnTpm3YsKF7Y69HeD8e9voIN27cuHXr1mmt161bN27cuHyXM2i99NJL//Iv/+I83rVrV2lpqda6oaEhHA5PnTo1HA7PnTt3586dea1xEOprD3PY58yaNWvOOuusdDrN0d5PXnzxxRUrVvQIo16P8P477I/4IAyFQh0dHVrrjo6OgoKCfJcz+GUymW9961vXXHON1vrNN9+cOXPmm2++2dTUNG/evEsuuSTf1Q02fe1hDvvcSCaTX/ziF9euXas52vtZjyDs9Qjvv8P+iL9YJhQKNTU1+f3+jo6O8vJyJibsV88///xNN900a9asO++80+3+1LcxNDQ0TJ48ec+ePfmqbdDrvoc57HPj7rvv3r59+4MPPtijnaP9sOu6g8DR6xHef4d97r5Zpp84d+KPHTuWO/H7ldb6lltuWb169dKlS8eNG+c0/uMf/0gkEtOmTRMRr9fr8/nyWuMg1Nce5rDPgWw2+9BDD73wwgvOU472XOr1CO+/w/6Iv1iGO/Fz47XXXlu2bNkzzzxTVVUVjUaj0aiIxGKxCy64YN26dalU6o477jj//PPzXeZg09ce5rDPgRdffHH48OFjxoxxnnK051KvR3g/HvaH8TRrXjQ3N59zzjnV1dW1tbUtLS35LmfQuvPOO/c9cmzb/tnPfjZ69OiysrJ58+a1trbmu8zBpq89zGGfA1/72td+9KMfdT3laO9XPcKo1yO8/w77I36MEACAQ3HEnxoFAOBQEIQAAKMRhAAAoxGEAACjEYQAAKMRhAAAoxGEAACjEYQAAKMRhAAAoxGEAACjEYQAAKMRhEDuqD703+bWr1/fT28ODBpH/HyEwJHlwQcfHDZsWL6rAPAJghDIqS996UsTJkzIdxUAPsGpUQCA0QhCYKBYv369UuqDDz44++yzi4qKjj322Mcff7xrqW3b99133+TJk8Ph8NSpU5988smuRVrrBx54YNKkSYWFhaeffvqrr77ataihoeHcc8+NRCJHHXXUb3/72672d955Z/bs2SUlJUVFRbNmzWIoESYjCIGc2rRp0/p9dF/h3HPPPeOMM5YsWTJjxozLLrvsD3/4g9P+05/+9Pbbb//GN77xxBNPzJo165JLLula9OCDDy5YsODqq69+9NFHy8vLzzjjjDVr1jiLvvnNb1544YUrVqyYPn36FVdcEY1GRSSbzZ599tmlpaUPPfTQww8/7PV6582bl8N9AAwwh3G2ewD7t/8fw3Xr1onIo48+2rX+9ddfP336dK21bdulpaW/+tWvuhZ9//vfP/30051FQ4YMefzxx532bDY7e/bsJUuWOJu7//77nXYnAtetW6e13rp1q4i89957zqLGxsZf//rX/fzRgYGLHiGQU04U9dB9hdra2q7HF1xwwXvvvScijY2NTU1N5513Xteic845x1nU1NS0Y8eOWbNmOe2WZf3xj3+87LLLnKennnqq8yAUCnW9tqqq6oorrvjiF784d+7cxYsXx+Pxr3/96/3yaYEjAUEIDFyWZWUymb4WZbNZEXFWcLlcva7WPf+6v7auru6DDz748pe//Prrr0+aNOmmm246fFUDRxiCEBhYVqxY0fV4+fLlU6ZMEZHy8vLS0tI//vGPXYtWrlzpLKqsrCwuLn7hhRecdtu2TzjhhLvuums/m2hubr7qqqtKSkrmz5//xBNPPPHEEz//+c/75cMARwLuIwRy6vnnn9/3Es3zzz+/6/GNN964c+fOSZMmvfDCCw888MDTTz8tIkqpm2+++ZprrnEWvfLKK/fee+/y5cudRTfeeONVV121Y8eOMWPG/Nd//dc777zz2GOP7aeGwsLCZ555JhaLXXzxxYlEoq6uburUqYf/owJHitwPSwLG2v+PoXOxzOuvv37qqaeGw+Gjjz666xIYrXU2m/3pT386ceLEYDB43HHHPfnkk90X3XPPPWPGjAkGgyeeeOJzzz3XtbnuQ5Ldn/7P//zPtGnTQqFQcXHxBRdcsGXLln7/8MBApXTfP5wAcmn9+vUTJ07kRxLIMcYIAQBG+/94dZbOM91OcwAAAABJRU5ErkJggg=="
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Demonstrates underfitting: training loss not close to 0\n",
    "# Also slight overfitting: test loss higher than train\n",
    "plot([lin[2,:], lin[3,:]],ylim=(.0,.4),labels=[:trnloss :tstloss],xlabel=\"Epochs\",ylabel=\"Loss\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlgAAAGQCAIAAAD9V4nPAAAABmJLR0QA/wD/AP+gvaeTAAAgAElEQVR4nO3de2AU5b0//s8zs/fdJLtJNgmJuQAC4aJSEG1BbmpRo0nU1tOLlFNr8auc1C89VuxBjV+PVAqeWtDjr/2KjVaOyvfgEVDxnFYu8dpa71EkWBElhFw22c1l77szz++PWZawhBAgmQ153q+/dufZ2XlmGPad5zIzjHNOAAAAopLSXQEAAIB0QhACAIDQEIQAACA0BCEAAAgNQQgAAEJDEAIAgNAQhAAAIDT9gtDn81VWVmZnZ1dVVfl8vuM/oChKeXl53yXbtm2bNm2a0+mcN2/e559/rldNAQBAIPoF4Zo1a0pLS1taWkpKStauXZtSun79+tmzZ+/bty+55ODBg4sXL96wYUNLS0tVVdVNN92kW1UBAEAcTLc7y0yaNGnbtm3l5eWNjY3V1dV9M4+Idu/eHQgEKisrk/Wpr69/5plnNmzYQEQej2fy5MkdHR36VBUAAMShXxA6HA6Px2O1WkOhUH5+fk9PTz+1Yf3UR1GUmpoaSZIee+yxvst37tz5m9/8xmAwpHy+urp6yZIlsVjMaDT2W5PeXn+b139uacEZ7A2kisfjsiwzxtJdEVGoqso5l2U53RURBedcUZTjf3Bg+JzSr4osy5J0mn2c+v2jcs61/dHOp0GutWPHjhUrVixatGjVqlUpRatXr541a9aMGTNSlo8bN05RlPb29oKC/qPu7x990vKXnWPv+JdT3AMYSFdXl81mM5vN6a6IKILBYCwWy8rKSndFRKEoitfrdbvd6a6IQDo7O51O5yD/+DjtFCQ9g7CwsLCpqWnChAnNzc1FRUUn/TznfOXKlW+99damTZsmTpx4/AdMJtPcuXMrKir6Xd1sNlssln6LDEYjIzpRKZwe7YAjCHWjqqosyziNdaMoygC/KjActAOuQytcv8kylZWVdXV1nPO6urrq6mptYX19/Yk+//bbb2/ZsuXFF18sLCz0+/1+v3+oaiJJRHjkBgAAEJGeLcLa2tobb7yxuLh4xowZGzdu1BYuXLjwRIOU9fX1+/btc7lcySVDNZyJUSwAAEjSLwidTuf27dtTFqZkW9+3d99999133z0cNeHE0CQEAACNiHeWYYwxPI4YAACISMwgJLQIAQDgCDGDEAAAIEHEIGQM82UAACBBxCAkhq5RAABIEDEI0RwEAIAkEYOQMGsUAACOEDIIAQAAjhAxCHEdIQAAJAkZhOmuAAAAjBwiBiExxjFrFAAAiEjMIGSMMQQhAAAQkaBBmO4KAADAyCFiEHJcTw8AAEeIGIToGgUAgCT9nkc4cqBrFABGmvnz5/f09KS7FiPCpEmTNm3apOcWRQxCYoxwHSEAjCQNDQ3btm3LyMhId0XS7PPPP3/wwQd13qiQQcgZYwhCABhZzjvvPJfLle5apJkkpWHATsgxQomQgwAAoBExCAmTRgEA4AgRg5BJuNcoAAAkiBiE2myZdNcBAABGBBGDEJdPAABAkpBBiPYgAAAcIWIQIgoBACBJxCDEg3kBACBJxCAkXD4BAABHiBiEDF2jAACDcPnll6e7CnoQMQiJYeIoAMDJ7dy5M91V0IOIQcjwPEIAgJO59tpriWj69OlExBjbuHFjQUGB9nrTpk0XXHBBTk7OunXriKi1tfWGG27Iy8sbN27ckiVLWlpatG9IWSv5eqQR8qbbSEIAGPFePsif2Kfqs63KEnbzpNR20datWxljH330kfb2nXfeSTYQDx48+NFHH+3evfuaa65Zvnz50qVLFy9e/PTTT0cikfXr1y9duvTll18+fq2+r0cUEYMQD+YFgJFviot+PFGnYZzxGSff0H333ed2u7XXt912G2Ps0ksvDYVCRLR79+5k8hFR8mMpa/V9PaKIGIQ5FrZPIYWTjKFCABipxmWwcYPIJ930zbCU5ya6XK6GhoZx48YRUSAQ6Ozs7HetkZmCJOYYoUUmo0QHetEoBAA4iVgsdtLPXH/99atXrw4Gg+3t7dXV1atXr9ahYkNIxCAkYlYD/8yHIAQAGEhFRcX48eNP+rFVq1YpijJ27NipU6eWlZX927/9mw51G0Iido0yxiwS/6yLqkrTXRUAgBFs+/bt2gve525cx7/OyMioq6s7fvUTrTXSiNkiJIuB7e0auf8qAACgGzGDkFlkdI0CAACRoEHIyCrR3i6uIgoBAIQnZBASSYxyLOxrP5IQAEB0YgYhI+JTnPRZV7orAgAA6SZkEDJGnE9xMQwTAgCAuEE42YmJowAAAxngMUyj6QlNQgYhERFNcbLPEIQAACc2wD2yR+bts0+PuEE41cU+843gKzwBANKq72OY6urqxowZk5ub+8gjj/QtOtsfwKQR8c4yWtdolokyjOxQgBfbR9BtbQEANJEvPgk1vKnPtsznnm89f07Kwr6PYbrjjjtef/11s9n8T//0T7fffnuyqLKy8qx+AJNGyCA8YqqL9vio2J7uegAAHEey2g25hTpty5458AfmzZu3cuXKJUuW/Pd//3ff5Wf7A5g0YgZh4sG82sTRK89BixAARhxj0Thj0bh01yJh69atf/7zn5966qnHH3/81VdfTS4/2x/ApBFyjJCRNjaIiaMAAAPTHsM0duzYsWPH1tbWvv/++32LzvYHMGn0C0Kfz1dZWZmdnV1VVeXz+Y7/gKIo5eXlp7TK6WHJFiEmjgIAnFjyMUwrVqz41re+tWDBgoceeqhv0dn+ACaNfkG4Zs2a0tLSlpaWkpKStWvXppSuX79+9uzZ+/btG/wqZ26ai+3BNfUAACewffv2gwcPEtGyZcs6Ozvb2tpuvvnmvkXaA5ja2to8Hs8TTzxhtyfmXJwtD2DS6BeEW7ZsqampMZvNNTU1L7zwQkrp+eeff++9957SKqePJfpGXWayyHQ4ONL/kQAAYPjoN1mmubm5tLSUiLRGXkrpwoULT3WVQCDw61//+o9//GPK8m9/+9vf+c53uru7LRZLvzXhvb1KPK71tU7KsP6tKTw/L35a+wRHdXV1RaNRs9mc7oqIIhQKxWIxVVXTXRFRKIrS1dVlNBqH6ftHfrNJN6qqar/PXV1dRGQwDCqnHA7Haf/r6BeEnHPGmPZCUZQzX8VoNE6bNu38889PWT558mSr1WqxWKxWa79fq5jNEUnSSqe6pC+Cpiutw3VyiyMcDlutVgShbjjnsiyf6CSHIacoitVqHb4Drv3WARExxrTjrB3wQQahJJ1+B6d+QVhYWNjU1DRhwoTm5uaioqIzX8VkMi1atKiioqLfdc1m84lahHGzmTGmlU7LURu83GKRT2VXoB/aAUcQ6kZVVVmWT3SSw5BTFGWAXxUYQsnfZ+2ADzIIz4R+Y4SVlZV1dXWc87q6uurqam1hfX39qa4yBI6MEZJ2KSEmjgIACEy/IKytrW1oaCguLt6zZ88999yjLex3aHDgVYbIkSB04mFMAABC069r1Ol0bt++PWVhyvhwytt+VxkKR/vi86zEGLWFKB9DLQAAQhLxzjLMZFZDgeRbXFYPACAyEYNQzsxmRnO847D2dioeVQ8AIDAxb7pN5nFTIl/u0e7sPhktQgAYAR577DFcD3P8JeM6EDQITWOnRr/cY7/o20Q01cWeP4CrkgEgnZ5//vlXXnllCG+qfPZasmSJzlsUNAjN46b630w8Q+viPPaxl/fEKBNX1QNAmlx22WWXXXZZumshKBHHCInIOGas2uNV/d1EZDfQJfnsT4fQKAQAEJGgQUiSZCqdFPlqr/auqlR68WsMEwIAiEjUICQyjZ0SPbBHe11ZIm1vUmNoEwIAiEfcIDSPmxr5MhGEY2w0IZO92YZGIQCAcMQNQlNpeaz5Sx6Lam+rSqWXvkaTEABAOOIGITNZjAUl0aa/a2+rStgWDBMCAIhH3CCkxNWEn2qvz8tmBkaf4hYzAACCEToIzeOmRA58lnx7TQnD3FEAANEIHYSmcdOiB/Ykn01YVSq9eBDDhAAAYhE6COUMl2TLiLUd1N7OK2D7e3hzAI1CAACBCB2EdOSmo9prmdEV50ivNCEIAQAEInoQmsceM0xYVcLQOwoAIBTRg9A0blqyRUhEFcXSG628N5bGGgEAgK5ED0JjfrEaDijdndpbh5EudrMdzWgUAgCIQvQgJMZMZZOjX/XpHS2VtuIiCgAAYQgfhNow4ZdHg/B746T/blK/6EEWAgAIAUFI5vHnRfZ9kHyba6GfnyevfBe9owAAQkAQkmnsFJKkyOcfJZf8fJr0joe/jYdRAAAIAEFIROSYd21v/X8l31pk+j8zpF+8oyAJAQBGPQQhEZHtwktjh/bHWg8ml/zjBCms0It4MBMAwGiHICQiYgajfc7V/te2JJdIjB6cJd/1Nzy2HgBglEMQJjjmVoU+flPp9SWXXHkOK3HQH/YhCQEARjMEYYJky7BOnxt4a3vfhQ9dLN//gYIbzQAAjGIIwqMyFn4n8NZ2Hosml1yQzS4vktY2KGmsFQAADCsE4VEGd5GxZGLw/V19F/56lrShUf2bBxNIAQBGJwThMTIWXN+76/nko3qJqMjOfjdH/uFupQcdpAAAoxGC8BjmCRdIFlu48b2+C68rky4tZLe/jQ5SAIBRCEGYyjH/2p5Xnu47UkhE674pv+Phz+7HDFIAgNEGQZjKNmOhsXBs51O/IvVoE9BmoP+8VP7nvypf9WKwEABgVEEQHocx5z/cTqrq2/zvfRefl83uPF/+3i4Fl9gDAIwmCMJ+MNmQc9M9sZavev70TN/l/3ye5DRR7fsYLAQAGD0QhP1jJnPu0vuD7+3yv/nS0YVE/7HQsOUrvv5TtAoBAEYJBOEJSfbM3P/1QO+rm0Kf/CW50G2hHRXyuj1q3efIQgCA0QBBOBBDbmHOT/9P1+ZH/a9tTS48x87+fKV873vqfx1AFgIAnPUQhCdhKp6Q98+PBN/b6du0jitxbeGELPbKlfKyt5VXmzGJFADg7IYgPDnZmev+2UNqsNfz7yuSj6e4IJttutSwuD6OB9kDAJzVEISDwkyWnJvusUyc7vnt8tihL7SFC8ewJ+cZvrMjfttbijeS3goCAMBpQhAOGmOZVy3Jqvppx/+9t+u//j816CeiimK29wajSaLyzbH1n6oKGocAAGcbBOGpsU6fm/8vG4ix1gdv9r+2lVTVaaL135L/fJVh8wF19ovx9zoQhgAAZxME4SmTbA7n9be5b1sdanir/bf/O/rVXiKansPeqDQsmyJV/Tn+v95UOtFTCgBwlkAQniZj0Th3zVrHwu90PvWg95mHlB4vI/rHCdLe7xotMk1GTykAwFkCQXgGGLPNWFCw8glDzpi2Nbf2/M9/8Hgsy0TrvyW/WmF4/iv1m9viuw4jDAEARjQE4ZliJnPmlYvzfr4u2vT3trW3Rf7+MRFdkM1ev8awfJpU87Yya2t88wG0DgEARigE4dAw5BbmLr3fWb3U+x9ru198gsdjjOjGc6VPv2O49xvS+k/VSZvjv9ur+vGYewCAEUa/IPT5fJWVldnZ2VVVVT6fbzClr7322vTp0zMyMqZPn/7666/rVtXTZpl6cf4v/6/S5Wn/zc9ihw8QkcSoqlR6s9Lw9Hz51WZ+znOxH+5WXjqo4llOAAAjhH5BuGbNmtLS0paWlpKSkrVr1w6mdPHixXfffbfX6125cuXixYt1q+qZkKyO7CX/knHpdz2P3dW78z+JJ7pEZ+ezFy6Xv/gH49wCtrZBLXw2dttbyt+70WEKAJBm+gXhli1bampqzGZzTU3NCy+8MJjSzMzM7u5uv9/f29vrcDh0q+qZs826PG/5b0MNb7c/fHvo4zeScZhrodsmS29cY3jvWsMYG5vzUvwX7yhd0fRWFgBAaIxznRolDofD4/FYrdZQKJSfn9/T03PS0vfee2/WrFnaB959990LL7yw7yoLFiwwm83FxcUpG1qwYEFlZWVra2tBQcFw7tAgcK78/YPIG9t4b5fpm1cZZl7GjKa+5d4oW/uZ8b+a5DvK4z8dHzOczSO2Pp/PZrOZzeZ0V0QUoVAoFotlZmamuyKiUBTF6/W63e50V0QgHR0dTqfTYDAM5sM2m81oNJ7ehga1gSHBOWeMaS8UJfUh7/2W3nXXXStWrFi+fPlvf/vbX/7ylzt27Oi7iizLZWVlkydPTvmqwsJC4xHDtTODZpz6TcvUb8YO7Am/tiX69kvmmZeZp8+Tcgu10nwj/WYW/8kE5a4P5D8eMPzLNOXaEi6z9Fb5NI2cYy6IWCxGRDjgupEkCWe4zrQDPsgglKTTb0noF4SFhYVNTU0TJkxobm4uKioaTOk777zzzDPPFBQU3HXXXWVlZSmrmM3m6urqioqKfjdntVptNttQ78Tpmjora+qs2OEDgb+92vuH++TMbOvMhbZvzJezcoholo12FdIrTfxXH0kPfEp3nictmSCZ5XTX+RSFw2G0CHUWi8VG0Ek+2imKEgqFcMD1pP2MDzIIz4R+nXGVlZV1dXWc87q6uurqam1hfX39AKXnn3/+H/7wB7/f//TTT19wwQW6VXWYGAvHOq+9Zcx9G7Mqb463fNW25tb2dT/vfXVTrHk/EVUUs7cqDU/Mlbd8rY7/z/i/faLiiRYAADrQb4ywq6vrxhtv/Pjjj2fMmLFx48asrCwiYixRgX5LGxsbb7755oaGBi0Ry8vL+35hRUVFTU3NiVqELS0tY8aMGf7dOn08Hovs/yT82d/Ce/7G41HL5FnGMWWyK8/gytvLc9fud+w6ELghu/Mfsj0XSB1qoNtcNtk0fhqT9WvEnxKv12u329Ei1E0wGIzFYtr/FNCBoiidnZ15eXnprohAPB6Py+XSoUWoXxAOubM9CPuKe5rDe9+LdxyOd7YpXe2Kz8MjIS7Lfnv+31nuQYO71J0xyfepzXfIMnG6ZcpFlkkz5KwcYiNoOBFBqDMEoc4QhPrTLQhHaPNCNAZ3kcN9zLgpj0W1KablRA1e/tx+9Tft/KvWrquD71/1+ruTtj1pCXWRySJbbJLVziw2yWJnVrtkdUgWm2TLkJ1uOTvf4HTLzhySzrbxRgAAHSEIR6i+F1qcn83Oz5aJSOW5+7oXvev59iMd/P0Ovr8jOM4QvDAzNNMRXOgK5rIgDwXUUEAN9sZavlJ87XFvu+rvYibzsd9sTsnOY0oNRtnpNrjyZJdbduURkeJrV3yeuK9d6fJwJS5Z7JLNoa0rZ7hkl1t2upkBU+kA4GyFIDybSIwmO9lkJ1sygYhI4Zn7ujM+6OBvtvJffKZOymI/HC/9wwwp19JnHVVRw6G+X8KjYTUc4OGgGgqo4QAPBY4pjUUVX3vw631Kl0fxtROR7MqTnW6Dy23ILSRZ5qGAGvTHO1t5OKD0eBVfh9LlkeyZPCNbyc435eTLrjw5O092uiVbhmR1SFb7UHXhqkG/VnOuxI8eE7OVWeySxZaS9wAAg4QgPIvJjKY42RQnW3wuPTpb/tMh/tx+deV7sW/lsapSqaKYlToYSbJkO/amPDaHTLlDWQ/OlR6v7+v9pnAvC3TFO1sif/8o3uXhQb8aDqhBv2SxMYuNyX1ajYwZcgqMReOMBaXGwrEGd5ES6FG6PEpnm9LlUbo7tMxTQ0EeDqihAA8H1HBQsjqY1SZZ7H1nDKnhIA8H1XCAVJVZbHJmtrGg1Fg0zjim1DBmrCE7fyj3FABGIwThKGGU6JoSdk2JHIjLrzSp2w/y+95XCmzs6mJ25TnSRW5mHb5/asbkrBzpHGY+wWSZ45txpKrxjsOxwwfC+z7sfW1L3NMs2bMMLrfsypddbkNBqdaUZBa7ZLVLFhuz2iWLfeBacCXOQwGluyPW8nWs5YD/zZdjhw+owd7EWKnLLbvy5AzXwDsiWR3MapeS/cYWGzNZBloFAM5+CMLRxm6gG8ZKN4wllcvvevjLTepd7yqfevkFOeySfHZJAZuTL2Xr24koWex0XIwZ3EWWybOGcCtMNjBHluTIMhaNTy7k0Ujc26Z18ypdnuihLwb6ClXVmrBaE1NrhvJ4XJt/ZMgpMBaONY4pMxaONeSXkCTxUEANB9WQn4cCzGKVzInAHpJOWjXoV3ztarBXsh6dBoV5TwDDAUE4akmMLs5jF+fJD8ykQJzeaedvtvFH96iLdysTs9hlRezyQmlO/nC2FEcAZjIbC0qMBSWn/Q1cifNwUA32HtOEbT1IRNqEI8nmYBY7j4SOZmf0mFshMNnALDbJateam8xo5tGwGgrwUEAN+Xk8JpmtyblLzGBQvO1xXztjkuxyS/ZMNRzgoaAa6lXDQWYwHWko27jRQg6XXDrBWDjWWDhWcjjP9GABiGpU/wrCEXYDXVrILi1kRFJMpb+0853N6n0fKA1efp6LTchi4zPZ+Awan8kmZjGd24sjHJMNzJ4p2TNPuwmr9dmq2mBnyM9jEWayJtp5FjszGNVImIcTH6B4TM7Ol13ufruCeSSkhoPaoGmoyxvztsU7W0Of/jV2+ABJssF17P2gJTkxPTgxZemY20gZsvONhWWGMWUn7XPWH4+E4r52tceb3Fk1HEwcw3BQDfp5OECSJFnszGKTbBmSxaZNmErsr9nKVVX780UNB3k4wEyWxF8tVhsz2yWrjVnsQzuTa/DUcFDxtfNY9Ojk7WPvxa8rzlXtD7JwgKsKM5oTR9Ui1p3kEITCMUo0r4DNK5Dvn0k9Mfqok+/v4ft7+MtNtL9H/aKHGxhNdrJJTjYpi03MorEZrMzBHLg+4nQl+2xP9AHZbKXMAQcvk19ltspmq3aL2ngwSH0uqFe6O5Ue7zGfVpTE9OBgrxoOEu/zMGjOo4e+CPzt1Vjr15I9w1hQmjIUeky6HBmjZRabduVMSqZKFhsN4n7HPBblkaNzlY++CAe1nFP83YqvTfF5uBI3ZOdJmdl9B2ulrBxDQcmRa37spKqJdAz5tRfxXp8We2okzCQp+RcAM9vUcJB3edRQMHFAErOm/WoowEwWJvftcGZyZrY27dngypOd7mRKqaqq9vaGPe6jlx6ZzImoDvnVUCCx9VBiE2o4mLL7irdN8Xk4Vw2uPGY0JWOeOGfmY4eiJYOkHXPtAqczCGseDR+ddBY9dgK5ovBoONFXYbExSeaxSGJHomHJmvoXkmTN0GarJbol+v7rWB2n9CeF1o+i/eukXHx15OgFeDjIYyq5LjzRlwwhBKHQMo1aKB5zBreFaG8X39fNG7t4fQs/0EsHerndQGUZbKqLXexm38xj01zsrH5o1OgjZ+VoAXlqOI97W+OtX/NYrO/iRFsqFIj7u5JtsmR3LtExt6PS4iTRIDOm9icc/bknOtpuszoS06Asdslql525UkGpnJElO92yK0+yZZzyjpwuNRwkte+fCKrS41V87XFfu+LzxPa9z+OJI8M5V6NRP6OjURoNJ1LBapcsR2d1JTO774aY0Sg782RXXuosbiIej6V0p3MlxpOt4aD/THaQmcxHJ52Zj23nSdIJW36cq8deWEXEteHwxF8wieuvgqq/K95xWA32nkKdOCW+KjGN7phzj1lskllr3NuVMeNpKoIQ0iHfSvlWtmBMajp+1cs/9vJ32vm/f6Z+7effyGHTXGx8JhufSeMzWLbCRlwXG5wUY4acMYacM70ZIY+EEr+PsdTHTGsXz6S5A/DEjk8CyZ5pHFN2/Ce1W6zlDsMt1pjB2M8tKQae4TzcGDs+sPX8A0Xj8Xj02RCCEAZFS8eL89gt5UREPTF6z8P3dvEvevhrLbS/R93fY88yUYkjXuJgxQ4qc7AiOxXZWImDCqxoPo5yiT7bob0+FUAvCEI4HZnG5OybBK/X2ys52mJyk58fDNBXfv5mGx0OqAcD5AnxXAsry6AyByvLSAw6Ftmp1MFsOAEBIN3wOwRDpsDKS53sInfqmLnCqTXIv/LTgV5+oJf+0saf2682B6gpwI0SnWNnxXYam8HGHZm5Oj6T2XFiAoBe8HsDw05mVGRnRXaak5+akb4INQf5QT8d6OX7e/gbrbS/R/2yl5fY2YxcNiOXzchhM3KZcySOLgHAKIEghHRymcllZtNcRH1miMdVauzmH3Tw9zv4tq/Vjzv5GBu7MJdd6Gazctn0HFzLAQBDCUEII45BomkuNs2VeMiGyqmxm7/r4e95+P/7Um3wcruBShys2M5KHVTiYAVWKrKzfCsV2lgW2o4AcIoQhDDSSUcesvGPExJL2kN0MMCb/Pygnw4G+Ied1BxQ20LUHOASo2/kHO1WnZjFJL3vHAIAZxkEIZx98qyUZ2UX5vYTcZ4wfdDBP+zkW7/ite+r+3t4ppGyTCzLRFkmcltYiYNKHazYQSV2VminHDNmrgKIDr8BMKq4LXTFOeyKc45mZHeUuqO8O0rdUWoP8yY/feXnb7dTk189HKSOMJcY5ZhZroXyrIkXORaWa6YiOxXZWZGNFdjO5BZXADDSIQhhlMsyUZYpGWT9JFogTh1h7glTR5g6wrwzTB0R/omP/tRMzQG1OcC9Ecq3sgwjJZ/U4TSR3chyzJRroTwryzWTy0zZZuYyU7aZXKZR/kwPgFEG/19BdHYD2R2sNHE/qX6SMqpSW4j3xih05NHCvggF4rwjTJ4wtQb5p17yRckbUX0R8ka4L0JhhbJMZJQo08isBnKZyWU6EpNmVmClfCu5LSzfStlm5jSjxQmQTghCgJMwSVRsPz6qThJeXVGKqdQb46E4eSPki3BvhHxR8kX4+x3UFiJPWG0LJYLTIFGGkawysxu1NmWifZllokwjyzSR3UB2A2WZmEUmq4EyjBSPkJ0xq0om3L4O4MwgCAGGhXYTALdloF7ZpJhK/hgF4zyQSE3yRrg3Qj0x8kX5V34KxCgQp56YGopTWKGeGEUVY3fUEIjHiMhhJKeJmWWyG4gROc1ERCaJ7IajG821UKGNFTuo0MaK7JRjZnYD4YpMAEIQAowERilxb4E+y07S4gwGg7FYLCsrK6pSIEa+KI8qFN8unRwAABejSURBVIiTyqk7SkQUVSkQP/q8JE+IDgf5rsPUHFAPB6kzwrVw1cY7M42UaaIsI2WZmMtMZplsBpIoMbzqMpNRIoeRbAZmlshpJuORJmyGEWkKZz0EIcDZzSSRKTVEk04++KiNd/bEqCdK3VHqiXFfhCIKBeOkEvminIi+7CUtbgNxNapSV4SiR5qw/jipnAptrMBGY6yswEa5lqPTiLQXuRaG/lsYyRCEAEI71Zbo8YJxOhzkrUFqCfGWIHWE+Wdd1B6ijrDaEabOCO8Ik0WmPCvLs1C+lZ1jpzyr1j1LdgNzmcluILuRHAbmMJIRkQm6QxACwBmxGejcTHZuJg0Qot1Rag9xT5jaw/xQgNpD/K1W6ohQIKZ2Rckfo0CcAnHeGyNFJac5McBpN5JJIqdJ64lljCVGXjONJEvkNLFED62BLDIRkVVm2gsiSn5Yk2UiiSW6dhG3kAJBCADDTruac0IWnbTFyYm6IhRRKRjn/hjF1KPzb5PDn91RUjn5olybZKRNICKikKJqL4iIc+qKHv3arihxToE4j6rUGyOJyG4kl4nZjeS2UIGVabcrclsoy0QZRqZN03WayWFgdiPhuWCjG/55AWAEYUQuc/JlSsmQiSgUiFNXlPtj1BGm1hD3hKgtxPf3UE+UemJqIE6BGHVHqTfGA3EKxsllJqvktBjiyblCfeflaheDalOKiMhlSq2t1kLVZiE5DCzZJJUlyjwy20hmlGkiIjIwyjCy5NfCcEMQAoBwzDKZZco2D+riFiLiRJ1BpcnjzcrO7YpoLdRj5uX6IkRHLoPhRyYZ9aVy2t+TmIXUG1OT83kVlXpiiddxlXpjR74nzpNfq108SpSYx6tddZppJLuRrDKjIzOEtRaz3UAmmcwS2QwsmbJaz7DdkJi1ZJITbVztm5O5m6Rw6okdswt2Q+J6G22asbm/vuWU7uizCIIQAOAktHaqauV5GYwyUkqGXVhJ3NVIJeqOci0vu6MUiHOtK9gkkd0gSYyyTBSIU1RJ9C0nkzXZMxyIExFpV9oQJXqV45x6Y2rfLcqMMo+NRn9c1a630aYZR4/5eEKy7zqFSUrcU0m7uZJ24wiTlGgfa5mtDdxqnQHaEiIyhuU5rjM8eIOCIAQAGNEsMiUnAeWcwhTfkdKrGlHIGyFflGt3iggrpI3+au3jiEq+KD8YoJhKXREiLfgVlYguzDDOKdWjhghCAAAYRmaZxthojE0L5lOIZ48nTGQdplr1hUnEAAAgtH6C8OKLL37uuef0rwoAAID++gnCvLy8Xbt26V8VAAAA/fUzRlhbW7ts2bIVK1ZUVla63e7k8vLych0rBgAAoId+gvCiiy4iovfee++hhx7qu5zz1CtjAAAAznb9BCECDwAAxIFZowAAILT+g/Cll16aO3dubm5udnb23Llzt2/frnO1AAAA9NFPEG7evPn666+fO3fu1q1btUSsrq5+4YUX9K8cAADAcOtnjPDBBx9csWLFr371K+3tnDlzVFX91a9+df311+tbNwAAgGHXT4vw888/v+SSS/oumT9//r59+/SqEgAAgH76CcLS0tI9e/b0XfLpp5+Wlupy61MAAAB99dM1esstt9x77735+flXX301EW3fvv1f//VfH3jgAd3rBgAAMOz6CcLbb789Ho///Oc/X7JkCRHl5OTU1tbefvvtutcNAABg2PUThJIk/eIXv7jjjjs8Hg8Rud1uxkbKc60AAACG1gmfPsEYy8vLy8vLG6oU9Pl8lZWV2dnZVVVVPp9vMKXxeHzZsmVut3vOnDnNzc1DUg0AAIC+9Hv6xJo1a0pLS1taWkpKStauXTuY0nXr1vX09Hz99dezZ8++7777hrxKAAAA/QRhbW3tRx99tGLFijfeeKOxjzPc0pYtW2pqasxmc01NzfGX5/db+uyzz9555502m622tva22247wwoAAAAcjx1/i+0T9YWe4c24HQ6Hx+OxWq2hUCg/P7+np+ekpTk5OUuXLn388cfHjRv35JNPnnfeeX1XWbhwodPpLCsrS9nQnDlzrrjiitbW1oKCgjOpMJwSn89ns9nMZnO6KyKKUCgUi8UyMzPTXRFRKIri9Xr7PpkOhltHR4fT6TQY+pnLcjyr1TrITx5Pv6dPcM61iOWcK4oymNKenh7O+Z49ex577LGlS5f+9a9/7bsKY8zhcGRnZ6d8lc1mY0cMx45Av3DM9YcDriec4frT7Zj3E4QXX3zx8uXLf/CDHwztlgoLC5uamiZMmNDc3FxUVDSYUrfbvXz58jFjxtTU1Kxbty5lFYvF8oMf/KCioqLfzdntdofDMbS7AAOIRqN2ux0tQt1IkhSLxXCS60ZRlEgkggOup1Ao5HA4TrudN3j6TZaprKysq6vjnNfV1VVXV2sL6+vrByi94oornnrqqUgk8vjjj1944YVDXiUAAAD9JsvU1tY2NDQUFxfv2bPnnnvu0RYuXLhwgNLVq1fv2rUrPz9/586dTzzxxBlWAAAA4Hj6TZYZchUVFTU1NSfqGm1paRkzZozOVRKZ1+tF16iegsFgLBbLyspKd0VEoShKZ2dnXl5euisiEI/H43K5dOga1W+yDAAAwAjU/xPqAQAABIEgBAAAoSWCkDG2devW5NLGxsa+I4Vbt27F1TMAADAqoUUIAABCQxACAIDQEIQAACA0BCEAAAgNQQgAAEI7ekH9oUOHkvdR+/LLL4ko+fbQoUP61wwAAEAHR4PwZz/7WUrZ5MmT9a0MAACA3hJBiNuqAQCAmDBGCAAAQkMQAgCA0BCEAAAgNAQhAAAIDUEIAABCQxACAIDQEIQAACA0BCEAAAgNQQgAAEJDEAIAgNAQhAAAIDQEIQAACA1BCAAAQkMQAgCA0BCEAAAgNAQhAAAIDUEIAABCQxACAIDQEIQAACA0BCEAAAgNQQgAAEJDEAIAgNAQhAAAIDQEIQAACA1BCAAAQkMQAgCA0BCEAAAgNAQhAAAIDUEIAABCQxACAIDQEIQAACA0BCEAAAgNQQgAAEJDEAIAgNAQhAAAIDQEIQAACA1BCAAAQkMQAgCA0HQKQp/PV1lZmZ2dXVVV5fP5Bl/66aef2u12fSoJAAAC0ikI16xZU1pa2tLSUlJSsnbt2kGWdnd3//jHPw4Gg/pUEgAABMQ45zpsZtKkSdu2bSsvL29sbKyurt63b99JSznn119//Y033njDDTf0W8lLL700Pz9/woQJKctnzZq1cOHC1tbWgoKC4dsjSOHz+Ww2m9lsTndFRBEMBuPxeGZmZrorIgpFUbxer9vtTndFBNLR0eF0Og0Gw2A+bLFYBvnJ453maqequbm5tLSUiLSW32BK16xZM378+O9+97sn+k7OeSwWC4fDKcvj8Tg/Yij3AQaEY64/HHA94QzXn27HXKcg5JwzxrQXiqKctHT37t3/8z//8+qrrw7wnVar9Sc/+UlFRUW/pX6/PyMjY2hqD4MQi8XsdjtahLqRZTkWi+Ek142iKNFoFAdcT+FwOCMj47TbeYOn0xhhYWFhU1MTETU3NxcVFZ20dOfOna+99prJZNICkjH25ptv6lNVAAAQik5BWFlZWVdXxzmvq6urrq7WFtbX15+odNWqVX0bxZzzSy65RJ+qAgCAUHQKwtra2oaGhuLi4j179txzzz3awoULFw5QCgAAoAOdxgidTuf27dtTFiaHQPstPf5jAAAAQw53lgEAAKEhCAEAQGgIQgAAEBqCEAAAhIYgBAAAoSEIAQBAaAhCAAAQGoIQAACEhiAEAAChIQgBAEBoCEIAABAaghAAAISGIAQAAKEhCAEAQGgIQgAAEBqCEAAAhIYgBAAAoSEIAQBAaAhCAAAQGoIQAACEhiAEAAChIQgBAEBoCEIAABAaghAAAISGIAQAAKEhCAEAQGgIQgAAEBqCEAAAhIYgBAAAoSEIAQBAaAhCAAAQGoIQAACEhiAEAAChIQgBAEBoCEIAABAaghAAAISGIAQAAKEhCAEAQGgIQgAAEBqCEAAAhIYgBAAAoSEIAQBAaAhCAAAQGoIQAACEhiAEAAChIQgBAEBoCEIAABAaghAAAISGIAQAAKHpF4Q+n6+ysjI7O7uqqsrn8w2mdNu2bdOmTXM6nfPmzfv88891qyoAAIhDvyBcs2ZNaWlpS0tLSUnJ2rVrT1p68ODBxYsXb9iwoaWlpaqq6qabbtKtqgAAIA7GOddnS5MmTdq2bVt5eXljY2N1dfW+ffsGLq2vr3/mmWc2bNhARB6PZ/LkyR0dHX1Xueyyy8aOHVteXp6yoenTp8+ePbu1tbWgoGC4dwqSfD6fzWYzm83progogsFgPB7PzMxMd0VEoSiK1+t1u93prohAOjo6nE6nwWAYzIfNZrMsy6e3oUFtYEg0NzeXlpYSkdbyO2npggULFixYQESKotTW1n7ve99LWUVV1cOHDx9/jAoLC2NHDM+uQD+0Ay5JGHXWSTwex0muJ0VRcMB1ph3wQbbWjEbjWRCEnHPGmPZCUZRBlu7YsWPFihWLFi1atWpVyipWq7WmpqaioqLfzQWDwaysrKHcARiQoih2ux0tQt0YjcZYLIaTXDeKosTjcRxwPUWj0aysrEG2CM+EfkFYWFjY1NQ0YcKE5ubmoqKik5ZyzleuXPnWW29t2rRp4sSJutUTAACEol9HVmVlZV1dHee8rq6uurpaW1hfX3+i0rfffnvLli0vvvhiYWGh3+/3+/26VRUAAMShXxDW1tY2NDQUFxfv2bPnnnvu0RYuXLjwRKX19fX79u1zuVwZR+hWVQAAEId+XaNOp3P79u0pC5OjoMeX3n333XfffbdOlQMAAFFhjh8AAAgNQQgAAEJDEAIAgNAQhAAAIDQEIQAACA1BCAAAQkMQAgCA0BCEAAAgNAQhAAAIDUEIAABCQxACAIDQEIQAACA0BCEAAAgNQQgAAEJDEAIAgNAQhAAAIDQEIQAACA1BCAAAQkMQAgCA0BCEAAAgNAQhAAAIDUEIAABCQxACAIDQEIQAACA0BCEAAAgNQQgAAEJDEAIAgNAQhAAAIDQEIQAACA1BCAAAQkMQAgCA0BCEAAAgNAQhAAAIDUEIAABCQxACAIDQEIQAACA0BCEAAAgNQQgAAEJDEAIAgNAQhAAAIDQEIQAACA1BCAAAQkMQAgCA0BCEAAAgNAQhAAAIDUEIAABCG7VB2NbWlu4qiMXr9UYikXTXQiCBQKCnpyfdtRCIoigejyfdtRBLR0dHPB7XYUOjNgi/8Y1vcM7TXQuBLFu2bNeuXemuhUCeeeaZ+++/P921EEhTU9OVV16Z7lqI5brrrvv888912NCoDUIAAIDB0C8IfT5fZWVldnZ2VVWVz+cbTOnAqwAAAJw5/YJwzZo1paWlLS0tJSUla9euHUzpwKsAAACcOabbQNqkSZO2bdtWXl7e2NhYXV29b9++k5YOvMr8+fONRmNRUVHKhgoLC8vKym699dbf/e53jLHh3i/QrF+/fv78+dOnT093RUSxc+fOw4cP/+hHP0p3RUTh8Xgefvjh1atXp7siArnvvvuWLl16zjnnDObDixYtGjt27OltSL8gdDgcHo/HarWGQqH8/PyUCW/9lg68yrZt2+64447jo66oqKisrMzv9zscjuHeKUgKhUJGo9FgMKS7IqKIxWKKolgslnRXRBSc82AwaLfb010RgQSDQbPZLMvyYD78s5/9bObMmae3If1+tjjnWmhxzhVFGUzpwKtUV1dXV1cPe70BAGBU02+MsLCwsKmpiYiam5v77c88vnTgVQAAAM6cfkFYWVlZV1fHOa+rq0u25Orr6wco7XchAADAENJvjLCrq+vGG2/8+OOPZ8yYsXHjxqysLCJiLFGBfkv7XQgAADCE9AtCAACAEWh03lkGV+LrYNu2bdOmTXM6nfPmzUveBmnOnDnsiFtvvTW9NRx9+j28ONuHDzsO4SQfHoqilJeX912i511WRmcQ4kr84Xbw4MHFixdv2LChpaWlqqrqpptuIiLOeWNj46FDh3p7e3t7e9etW5fuao4qJzq8ONuHT28f995771133YWTfDisX79+9uzZKVeK63qXFT4aTZw4ce/evZzzvXv3Tpw4Md3VGYV2797905/+VHvd3t6ek5PDOW9paXE4HDNnznQ4HNXV1W1tbWmt42hzosOLs10HDQ0Nl112WSwWw0k+HHbt2vXSSy+l5FG/J/Ywne2jMwjtdnswGOScB4PBjIyMdFdnNIvH47feeuuyZcs45x9++OHChQs//PDDzs7OJUuWfP/730937UaVEx1enO3DLRKJXHTRRXv27OE4yYdTShD2e2IP09k+OifL2O32zs5Oi8USDAbdbncgEEh3jUanHTt2rFixYtGiRatWrUq5p0xLS8vUqVO9Xm+66ja69T28ONuH2+rVqw8fPvzoo4+mLMdJPrSSFxFo+j2xh+lsH503xNKuxJ8wYQKuxB8mnPOVK1e+9dZbmzZtmjhxorbwgw8+CIfDs2fPJiKTyWQ2m9Nax9HmRIcXZ/uwUhTl97///c6dO7W3OMl10++JPUxn++icLIMr8Yfb22+/vWXLlhdffLGwsNDv9/v9fiIKBALXXXfd3r17o9HoAw88cO2116a7mqPKiQ4vzvZhtWvXruLi4nPPPVd7i5NcN7reZWWo+lhHFJ/PV1FRUVRUVFlZ2dXVle7qjEKrVq06/kRSVfWxxx4bP358bm7ukiVLuru7013NUeVEhxdn+7D64Q9/eP/99yff4iQfPil51O+JPUxn++gcIwQAABik0dk1CgAAMEgIQgAAEBqCEAAAhIYgBAAAoSEIAQBAaAhCAAAQGoIQAACEhiAEAAChIQgBAEBoCEIAABAaghAAAISGIARIA3YCw7e5xsbGYfpygLPd6HweIcDI9+ijj55zzjnprgUAIAgB0uTyyy8vLy9Pdy0AAF2jAAAgNgQhwIjT2NjIGPviiy+uuOKKrKysCy644Nlnn02Wqqq6bt26qVOnOhyOmTNnPv/888kizvkjjzwyZcqUzMzMuXPnvvnmm8milpaWq6++2ul0lpWVPfPMM8nln3zyyVVXXZWdnZ2VlbVo0SIMJYKAEIQA6fHll182HqfvB66++uoFCxZs3Lhx/vz5ixcvfvnll7XlDz/88L333vvjH/948+bNixYt+v73v58sevTRR2tra2+77bYnn3zS7XYvWLCgoaFBK/rJT35y/fXXv/TSS/Pmzbvpppv8fj8RKYpyxRVX5OTk/P73v3/88cdNJtOSJUt0PAYAI8NQPeoeAAZv4P+Pe/fuJaInn3wy+fnly5fPmzePc66qak5Ozh//+Mdk0S9/+cu5c+dqRQUFBc8++6y2XFGUq666auPGjdrm1q9fry3XInDv3r2c86amJiL67LPPtCKPx/P0008P864DjDhoEQKkhxZFKfp+oLKyMvn6uuuu++yzz4jI4/F0dnZec801yaKKigqtqLOzs7W1ddGiRdpySZJeeeWVxYsXa2/nzJmjvbDb7cl1CwsLb7rpposuuqi6unrt2rWhUOhHP/rRsOwtwAiGIAQ4C0iSFI/HT1SkKAoRaR+QZbnfj/XNv77r1tXVffHFF9/+9rfffffdKVOmrFixYuhqDXB2QBACjFAvvfRS8vXWrVunTZtGRG63Oycn55VXXkkWbd++XSvKz893uVw7d+7UlquqeuGFFz744IMDbMLn891yyy3Z2dk1NTWbN2/evHnz7373u2HZGYARDNcRAqTHjh07jp+iee211yZf33nnnW1tbVOmTNm5c+cjjzyybds2ImKM3XXXXcuWLdOKXn/99Yceemjr1q1a0Z133nnLLbe0traee+65zz333CeffPLUU08NUIfMzMwXX3wxEAh873vfC4fDdXV1M2fOHPpdBRjh9B+WBICB/z9qk2XefffdOXPmOByO8847LzkFhnOuKMrDDz88efJkm802ffr0559/vm/Rr3/963PPPddms82aNetPf/pTcnN9hyT7vv3LX/4ye/Zsu93ucrmuu+66gwcPDvvOA4wwjJ/4/yQApEVjY+PkyZPxfxNAHxgjBAAAof3/+vJe7g19qaAAAAAASUVORK5CYII="
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# this is the error plot, we get to about 7.5% test error, i.e. 92.5% accuracy\n",
    "plot([lin[4,:], lin[5,:]],ylim=(.0,.12),labels=[:trnerr :tsterr],xlabel=\"Epochs\",ylabel=\"Error\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Visualizing the learned weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjAAAAA4CAAAAADGVp33AAAABGdBTUEAALGPC/xhBQAAACBjSFJNAAB6JgAAgIQAAPoAAACA6AAAdTAAAOpgAAA6mAAAF3CculE8AAAAAmJLR0QA/4ePzL8AACLhSURBVHja7d1nlF3lnSb639lnn1S5VFXKOYJyQrJIJksIMMYmOKeennHb3cbT4x7b3fZ4HMa3p9v0ctuept2OGGygsTEGbMAEgUAIJKGAco5VKpVUOZ187ofa5va9n66+17NWaZ1VJW29+931/vc/Pk/sq5IKAiFKKIkhLi9UkpITKAsFiuIoIlBWERNXFEoYFlMRqqgIxA2pUlQUFxdXkhMKFKSUlMSjv1sUKMvIiqvISYgjpiwrVJFQEigKif51IVpZSklRIKGCgqSSklARZBRUxJURQzH6XBFTUIUCqIirIKYiMCQuJVCMdiCGUAkxsWgFIzsRlxdTQVFaUVkMJQkFgSBaUUVcTElMUagQXaMsJiYQV1KUl1EWKImLycqIKalEu5tUVBKIiSkJVFARKCtGqyAhiL5bkIjuceQKI6sfWVNFTExcICuGhMI79x1TRCimrKIsUFEWF1NREec/PPOR34NRjOIiECZQlpBTEQgEytEZikuqiCkKorMycvYqYgIVSQWhikJ0FoYQIq9Gk0aH9QnFxWWUo7MxclpCOXGBioJ49Lsdl1BQQVmVnFBeqCgpFp2XioScQKjTeTFjJPU6YaoWR5RMVW9IKKckJVQWk1SOLOPIfVFWlpBFQl4CJQWhuEApOnlxZQVJBRUZeY0aVDknLmaOVq3qtKuSFldUEBo2pCStLCuDMkoCZYFACQVxobyEnKQRq5CJfhYqiUvJi4vLiauoKAtVQD76NGIRKtLRaS+qKIGyUvTsioJ37HdcQVZSXEFCTjzaz3xko8oCBWWBsoSSmFoJRQNCY/XrUhFE1rqkElnJUYziIhCWFKM3HZT1ykspaLFCn4VOO6RTSn/kSwSRrQkNK0tHngFVJuo3VsJ5FeMiD6PVFENiUsYo6xaL3vIpeQEycioy0hK6FfUIpFxQIy2mLB7Zt4JE9P4tKynqljPggNAx7WY5a0BOtbSMmJiymnd8lOrozJKLrtKjE2UFtaYYJ9DuPJIC9e/c48h1ypIS5hqj3VXyktoknFAWqlajA/1O2ikhUBDKSUbnvxR5ewVFBSlFJRU1BlSjqKyMQmTBA6Fi5LuQMKQgoSCnKJCT0axHWtyghIRSZBUqYkJBZG0yCvKKspIqQnkxOanIz0oqiAlUKSkoSknKySqLCwSGlA3qklUtqVpSXFxSWSBuxGqOYhQXgTAevSfThsUNyDvnpHbrDGl1XpOdDrhUVkmgRqBJWUVeSkU+ip5aNPqQrLQ2Ex2wxzQVbSp2aTYsLlATxRsFGSWBOXLq1DlijhuxxSqdzgildZjkO85o0KwmOvMZ2SiqyYk57JQaA9htNyiqtUKjanExg5KRJ1AUKglcUGtAxZBQlyWyhszTiTnmmK5sj52qVOkTE49Obl7KBFn9mhyTdlSLcwIv6XWdqZ60whSXKXvTGEEUrQXyClIGlOT06VKjICWQMaQiLishEa1sxGuKRzFKXFlRXlIoa8AER9WIG2OifWotUiumVUUhihODyJbBkIqcrC7dcgbVGGeSDjG1UpKR15QXE4pLKmpTkDfbsFoNel0ldEKdvBcNanTWOHVqBUYtzCguGmExitHL6gXyJsta5riydQ4a0qPOVGudMOC4HgddarqkUFEZPca53JCYkiOKaj3mWiVvY56tGrzqjImGLVaUkpOUV6PKcoGcS802U16vVbaoNcUfXOWoDQbVedOtKorRGziFlGFZk3Sods5YYy120DkrjNXlNmuEis5p9aYeSRl5cUVZb8uZaZMJrvN55/yL27zfM5qt8KrP+JYuHzHH6zaqSCgqoEbcMXkNDpthmy5jJR11k5keN1OTbuMNeViTHvWRlYCyHu1yTjqqrFlMv5yEOmNNlDFVv4yYjEExCXklobJegzJmWaZH3CJPuFeXlG1ibrDAPhmnLNbjrMHI+yjLRZ7QeZ2KzusW6FfSZoIaHWarNU2LjKQhGaUokssblDRRjSVaLbNBxQbLvewOC51TUG+8AUeM0yA+amFGcXGIfUNMSUmDaQp2eFZeowMmuNmQ2z1sv6uljHWdC07Z4ZBL1SqLRVnXBjFMs0ODNU5FOYmr7dPthDn6HbJL0RoJSYGK0DTzrTTshC2qxJTlpRwzzTW+bKE5NuoWt89MkwVS7+RrG2QcNaxsssOqzHS//6QkpahJs1ZZ73KFn9ipWirKgA54Ta9e18qYZ7k9DprsuMPeY4VOL/mSp6wUc5nzfq1LQUlgULOtJjnremfM9BmXO+ljnsM680yyS43dPmWDg6YqGxZGEcilyuJm6/eEUxbJOaRfwlaBHsw1U7PJQsSijG1Fv2PqLNBtvid8WcUb5jusxc0+6g7T7dLqqBvM83rkQxLIGzKo06ALJvqmI8b6O59wr9UqahWcd9Ysc4zRoBDlaYoG7bVEnQl6rLHPm27U5JScOSbabLMpamwwaKzxakYtzCguDmFZMarovClmiz4Fs/y5V72BTeqdcK1NbvG7KEtRIxl55FnVLtUjFNhtvgl+ZLxAzEy7vCntgkOSur1qhqy0IKpU1er0tEk2GGeeeq/b717HJRwy1h0+a51LHTJXjWH18lFVo9tsUxQN6NOg4C2HFHzXOU2W2O5DegybjryMBjlVSkIHDVutbL6S046rNcsBH/aCoy6Y6F677DNbDC0mGXgnr5kV2mKRfc77vss02GeDlV6z1U/NNM4YCQc0azeoFgV1ZjpvnmmGtHjTpy2w2y+tNs6wv3BUt5NWOWy7pEbVEGXAh9Tpcc4YJ0z2iM+ptdUHPWqnAbd6VJt+X3BKo5E6WUpZr16bpCTMdodWPQrmeMReRw0JPKnZm152XkZaIJSIqki9lmhU67jHdetx2hQFb1uuVUKfhd6jWqsHVFtt+qiFGcXFIRzJIiYE9jlkihNWqfW69V6yzANmS9jgJuMc1G+v5ywyy0jFpV5opZ2mesjtntThZi/6vN952uUe8wkD2qyzxjOOKyjJyGlxWrvbVHlNk6T99hpwk6/6Wwf80RLf8FHnfcPlziuZp15aQVxZrQFTHBfI6XfSUj80z7ClWi2SkpFUb0C7aWLSjisL5JTsNsbt0q71tCvtV3LUgA3+0jklv9TkaoFpoE9ORUVFt5TxtrjOKSx3pSNK5unXrlnaEactc1Cn9c6YGOVYqi1w1A6hSb5mnZJfa7LeTy1To8llXvNJh6WlDMtLyKigpCSlXlncVR63xGM+Z7l/8yEl3/JJ33XKx2w13TQb1cmKG5TUZ4f93u2sac5gl9tM0OkmvOiEagljzZdT0Cmm8T/UsKuMcdImn1OxQKNzDkjY5VN6jHHC9yXcod9ThkZ9mFFcLMJ45FUMCqSEJtum2Xn91sn7C4/6nx53wQZl1V5AJaoEVwxJadZjm6k2y/kvDur3A3MsstMX/NSDNtvpdf/Nfc5qMiTlvJSYWi/bodEfnfdxh12wyqvy5jnnb2yS8Wc+ZQrmKaqoGFStpMs5BVX2eI8ms11vWJdGex2z0lumaPCQaiWLPSGUVlCWQ68ztqkx0T6Bo5b7jZmOyuu3xAq7bHcDmOi4ikBWSo1uE22RNlmvgzrd5rfOa3LaQTRoc8rbXjJewTQpdeqMUVEtb7wvOG6D2431DZ8x1xvOWG9Ih9/4lGp/VFb1Tg2pXoNa79ftNdfKu9LlJtnhO+5yVJ3LVOnT6LSVlmm3W5usHntkrXSbn3qvX2DAWt+z2G+FdrrUXtt8Rb9WgWHEhbIqUoZ16zNknQWedci1NrhTzBpbnLLflYYNuly3X8sYHrUwo7g4hGUj2dNz9qgyQUqTsRbpVG+8/2SOtwQ26TTdUcyVURKKKek20Sbr/bkagWsMeN58CdWWmmWPz3tU0inT9SEpbaTeXW2OekkLTTRTj10etlTKl1zipC1+b7VGezS43otSwqiThd2KijpUmW6b83pUW2iTbv9gk/HmmmXYZY75rVlqtCgIJEBco1XKXrfc9cZ4Xrs3/NZd5njNMfc6HGVtaqJKeUJStX0SVjhsl5Oul/Ebiw3p1Akmm2aPFgUHtZglUC1UUeuU0OPyxmiz3V0WWKPVeLfYZ7mHfELJWeOU1GhwzrCKglC1QYcsE3fYXA9ZLOn/KLrez92IirKpOjU676AaJTmBYetdj1dNcr195mo15Dr3aTQs9H5fsFKnQFJZSVZFKKVJ2gnDOlS8229tca9HPeArxptqhWMeVvExN6pWGe24G8XFIoyJKeozLHBI0VxJM0wSc7sFuEzCVlMcUyOnCZOiDtmEIQlPWeIx35L2nMWukvCIW/zQLLd40UmTzHUg6sKrMqyoIm+ubV40zS67PCUj5mk/8ZaXLVejRZeyfWa4yhEZFSUVca1KYl414F0qLhdK+5z/hg/7uffqcqXfWCrhi2aqMkVCSUVetdvVOuIjJrtZmx0WqlhgkTf8SkaN+QKd1oAVnhagX8kRobTxGmxwlz163OaQQVWK4mLq7DTgbj2ekdCvVsEUaTRq12are5T93Df82OtWm+R3xprrn5xyznZ5c8V0GbHz7T5um1OmaTXDnQ75qvtd7lmbHPd5s+2TsNw5DzrvXcYqSGkXM94l+nRaoMtOgYxve58NmhxUY7x2K9WYYLM14lG8Wi0n4xJ9htSZ6l992G5bTbHKgDXO+q5V1pqHYtRjPWphRnFRCENloYxm8/WgVc5+rTrVgHOe9R4vus6weZJiUmJRZTTtkE5PKtvkeTwSXfQFGbe7xSd91bctd8pRt7taq0o0MRAX2uVNrwklZKw3WdIUFQ/6ukfVW+CMffZqibpnynLa1DtmrTlazbbO3b7nOjvxUR92xDP2OqldLY6ZZIwAeSXdLtXkamdUvKCoxVYfdJ+3TXSzqV7wghfwQVfgsLiimHrtMpbqNdEeX7HRFJ9Q799wt8dNt0yXGx3ydQ95xlELBZIWqZay3SlzxcT8lVOed9qd8lrNtMgzFmpVK+u8MVJiknJSJrlghVa3Om3QUW1KbnelPoed9z2XKGhyrUW+5MfmOI+CVn2uMN4F833Yy1pknPI1p9ytYp4TNvmSo/IelDSgJarN9at2wFFXqih60Gyn/cRdfqLXz13wtGuMs9pBm8w0ScOoDzOKi0U4UjOpUS/vA2gz1vNafdVh0xSdsMKgC8aaKRBGvamZqK/3hHYPR5eqNgimyjnnjMlecoNx+nTY5idusNtzCrJinnTGoGNWOetqH/SASY74sJ+5yX1W2+M3HvC4meZG1dy4ooQzDrlRwaflPOke/wXf8zmvudSz+s3zUdd5SY9LDMpFPWkJizQLLfa85w35rNc12OWA9f5Fu4X6LbAXV+L7LlEm6oQbNM5Kc9TI4u9sVdZnnHp3Ou4JNxjvXo3+s0cMC8XEJZBQJecKH/XXHnCj/4Wv+6LJalzwsu3GWWpQaE40u5WWM1m7y7zicb/1Pv9uin6f8IwmA262zcd1eJdeGRlTbFNAWspYnzZWi2c8Z47TQp/1cQuNc0SrWnfa6Grdvuw3BgxHEyFFQ8YY7zr/rsNMDc76pqxe0GGNudr8ylKzjHeVQZVRCzOKi0NYEFeRkdIk0G65k44ZmVWiU7Vb3Oq0Rv1KQhUlY6I4iZRt0YUWCS1woy1udNRPNVviuAd0WudSl7oZyx21XyDUb5ukJiUrxP1CTtzH7PSYuzRrlvUL9MtqVIpmi0K1er3PCivt1+zdSOGf/Y21nnGtGnutl7TGLjtcL29YICEw1lRTfNGfOWqKDVZaZYPAw4653gwfMOQPeu3Cr1wvp1ZJWU6/td7rPu3GW+s7Ljco48s6vGSZj3jZUuc1GicrY1CTbgcsdkrCPZ6103HXegqU3G+xHdrs8DV5L2qyAFnDEggMOCIl4YAJ+vRa7xL1nvL3nvBvbrTZVT7ovW7UaVCPFOJaVFmizw/cpN0DZjgsboJ6XZY6os11ahy1xw8cMsbMqDYXiFmhVcYNjita6BU9Ph890VaL7PekFaZbgrkOjPb0juJiEY5E1qGKwHntLnEKtIz8WMz/tgV9EsaoVh/NDFeM5G7r9Vrjv3rGoJlOyAqFVqmxyb12WOM1La51XouK42JRp3uTF4T6XWqKTodN9IhxbnJAs2O2W+o/e8QSkw1JSEiIaTLW32Oz6d7GVpeo1WSiZ602xWM+5TfOmS6v7JyiSVGHbULOE3he2hir/cwm3WpdoqJdzB98yBj32qPNs4ZVRRPGVTZa60VDrjdVgx960BumSVvhVxKOqHZGSqcOH7dBjbyko3p1y/iRmDpD3vY0OKqiwUL3Cp3xY0vc7ufOapEy0pHf66BqVf7BbBvcabUDOtzt1wbc7pCKj1rmc+qtsF5gZAa8wURn/dINttqtUZ+cJQJbrEDGBT+01CFHvIAZ6uUkEDplwMfsc6teDZ6z0HSf9q8gZcBxnzPTQSullaVHe3pHcbEIRxgT+oVanTJJQTf+zGSX4cf+yl3giFpD4hJqxJCU1+9yd1mj4IdmaNGmDgUp83Vp8YTL/cxqv9fubsRURxwFedO9X84Br5gtboF9KprU6XXGPne61QFrjXXKb7TICwQa1drlqCadGk12WrM/KkhqNMWAdpt93wzvFrNFwk3KRqap8/pst9xm16vyplTUAxI33xH/4jEnrHKfv3a1swajfENg2CGbfchaoQ3Gmq/J9eZ6zC4fVFRyuZT7rZZ2jZeVxQzYb7q3dGi2yyHMV8LlXvA9f+MkXrTcx9TY5ZROKbWRJYxJe8xEaxXcYK9/tNaAd9vskH5XOKvkuFvtc0ynrISYhIQ+L2q2Ud4CWywzzVFvOGShnPFOaPa2LZKgJCsT8UcMu95k33aTBkW3+7nzkX1ht0lSrvCMSQY06DAsNWphRnFxCEtRP0ZW1hl9ut3oEln3W2CGpyJfhnESsnLGGZARKChK+IjAYY3q/MF/VyNpjdle9wdX+pZ/lDLFfT5gILpKRkVFixrXecWTKk77iLNqDftLD9jiNbd5wJCtDrrSDDP8SL2YYTWytvuu5S7zEdU22ClvkRu8y3KnnTfFSd9x3Pf0WiButwVKyobUOWy6Hq9I+obPOK+gXs71HrHWh33Ejba6z3nbhIakZQQYkrHYRAXdbtNvkQeEitKabfRXarxgkjGOm61VjawGJcM2mmufWf7amw74Be4w3evSPuGEl832lh7PO2tIhmgqe4RTIWaqReJOu81JN3vOdim3OOw1L0k6Z9DXPWufCcpISsnKynlDi8v81CQT1NljsQ1usVe9f9PiTe/DCC9ORkVBoEFC2vu8aaYLDlvuJ2CNmPWqNevSjDf9wW7jpEctzCguDmEoL6uk4JjDmk13yBq/91EtuNY3MNk8Gc2aDUo6Y07EKDJGpwZ5/cZZp+AqPcrabHOX16x1i35LPKjKZtep0aVHWVbJFZr0mWiGRv8X0i61w/Oafc56/+xb/slyJWd9ywlTJKLqx37T5ey3WZ2SUJ89rrbcLnFZ433TFWbrlFbRbZphNSpKGiwzx3rH9bnCq2o8LKFJwlcsM85zjqnzO6vVypqoXvydjrQOr8pYrtp0zNaiRruyj+kQN1vRfISSxkgoKEgbNNb1NntBrZK8Na7C92W1+q6l7rfWTjPFrdGuYCjixsipNkHaEh2u8qwbnZRwxjM+67hNPiCQ0Gaiu91vopScvLxOP1JvyGXOWKbWy66wwGGf8oQ6/apUme950CiUF0hEbA5nTdPux+5xh3+3ysvocqOVnrTcgMNWOKtPs0Q0YT2KUfz/RhiL+IwG7Ue3Swx4Tsoxj5rlCSfd6d3azHFWq2p1VukXGJRwwWb3WG2jj/pHb5jsdZ/2sgV+5Rq32mie580w2fek8DuBhBwmOmylCR4019totkPeTP9suT+Y5w8WWKvWr71uvhgoy5ttmkCtvcZLe9Ra95hjm8Mu9QXzzdOozVOukJAxXrWiuKSsTzot7m+94qBmc42zQIO8NhPdZ71WFTPUavaajLKylIqMmV6xztXeNtubdvk7F3zfZJfJavJrgbTTFhuSN0VJUiCpymEDntKoSc50szzmDgdVtJvmoIm2i9nsj86bpzZi1Ys57oC1pnnDZI3eJ9Smz9tu9CsnfUTePA95jzqb1YpFk9xpnU474ovO+b2Jrrbch433bW+7ILDPOHNdUMIUCYmoJpQ37Kh2VSruUKVTzoto9m5NXlLrNR1mGafPaRUpA6MWZhQXh7CsKCmvw4AJPu2kd0s75A1L/MBnXe77mqxyWKeUiaYZNCwvpqRsg/lO2WKW+aZJGuMxH/CIT8pJafKEm4x1g00aLDDVZnmBPchZ6GWh0HulbPcRU6VsdcJ1Dsk65aR9GiyNWJACRYFbHDBXxVGPOyJQ5XJ3+6Jj9mmSt90brpL3ljkmSCqrqJipWouNahzylx5QLSM01UZX2aHDdU67Q5su4+SNNyStJGVYySVa1Dtsr1WWuxWv+KK3dai2xSXOmmWcNgd1qtOoGPF0djigRZtF4v6o2XKvuNELDjvpgI87Zpcu1Mu9w4Q55IKE99rjoJQTcp4311YVp610i35Fu81yh6SHlaRkJeRRrcZ4h8wzxwt67HCrM/7BkPl+Z455ig4pWGqRuFzEp0fMoEmmq9Xrae+yxE99QZ9jdrtH0WHXa9LgiKSsUX6YUVw0QlH834y7BO502jrrDXjOPG+7yV9Z7Iwj5jmvxxg9ykoSYppN0aTGQpOckHXBe230Iztd44RX3OKvlaVsMmSlvfYpS0bvz4X2+6ynFSxwvwYP+bz5Nqu3wxybfc0TBoyXVCcUU1GSdA4PSnpd3mRntLreSyYZa4lZ3pJVJ2epAWOj+lOgKCvjt9bYao+rrPMHk5xQL+m4D2lxwCS/kfY1L3nbLGcQk5WQEThlwGc876zptluuqMpM7XZY7yE3OW6DGbrUqlUSGhQY8KoLelBwyDTDEpYr6XRcAT+LptOnqlGUjDj1Av0S3pD1A4/5hT/Xao4b/Yv3+a4lUhrdZqIewwZNjPhkkkoSFlput39S54gj+Fe3G5QTd4mjWmxzxkrzLdejX15CXCBlptXediDi6ckY5yseU++MYYfV2GeFE46oVx9FcqMYxUUgLIjLm2yV+/XZrV2Ha3xfVsH7dfqhBi12uE67b2k2W0qvmLwqtartQOBFT/qKAzq9Yq1av/UBtxu2xP+2wGJTtTlhSEEoMNdJWT90t5SKO63R6L/iNYs0eUOd63xBnyUec8FSMXlJKWVnDJnhl/h7HU5oUjHfaRP9zLVOWScpcIuSM0SctXHnlC2zQ9pntOpVZbzpOtyqy0tWOKHsdpMUVJnlwjuMuSUJRV12mOUvDXtRFe7BCWPdpNd07TZabK+cJnEJJXHnDBmOKjcVoaXOi3vFaXN81m+dk422vxfVEWNgTsIkE3S4VY9jvuCrVnvCvdba6RLX2OKCXlstd0SPWZKRHxKqNqzPRMv1ucwFm43zhqzJej2jCvVudVKDZiciZuWKnBoHNCpq8ohxGkzSZYtQ0lwPq9VliT47jDEklNY/amFGcXEIR87BBTNR55hutf67V/SoFTNL2U1qTRV6zZC49ogNPKGgJGG8dpcpuDmKtubZ45zQPi+60Rar5GVtN+CUC+qNTLic06lWqw4n3O2C2z0iJzBOq/Vecs5ar5usVVleWqgoriRpULvlsv7ohFkSTlrloA0ycma4xrM+YqND6pUkiZQA8vilbtdpM9luEzSrVnHB5Xa4VZttdphvug5ZaXklZUkpk8Ts9ZaUZWZ6WrOYn/hbz3lJTrXQSs/pMkcq4k1nQNF4S3W6whGhifZZ4Nc+6IJGdaqN8RpEkz4jf4YKZquS0+68Nj9yhdWm63S543rs1uKwh3xIu6SpGiMWzJFMdsVTZhtwq23udo2M31vnPgNWOeMNH5L1PrP0SRuxe6QN+p060+z1bqE+5wW+4EFVTvgLL1mp3VFL5AxjeHQuaRQXi7CMtPOeNl9BuwsWaPYBn/aojdYqCvwv0x11SJVuGWl1hsRUnHHYZMftc6l+9bZ7r2oPWSYnr2KGdXqMETjnlISEgri4irFWaVawwEOWKUo45H/6ke3+6C5/7jHdsh7VaGLEUh5XUFajLLREStI6Z6XEdVmtzc2OudIZ79LlJRXVpkTxWElCWrfPeMXTvuyI73hc4GHrtVso65NWWueYpD5ZA4ajeCAr0KJWne06PO+AcYYFxviuo9JWC3VpM83USN1gSCBQo9Zis803z1wLndDlK2rts8BmU52I7Atx/VKSEkZ8kax2r3jYDN/2qnYHTXKTR8yw2CvmWySn4qxJMmKGeOcOz2pxUoNaG4V6jTFFtdVSajS71jjLXGO//QbeYdQLJDR4Tc5cfVaZqWijjZp16zXBIZuVrHJMSVpcfrTjbhQXi9g3FcQM6dWo7LgDlmp3WlKfKs1edY9litpVVFSLSyOpIi/ukAF70OgSUx1X6yonfNS/O+8mU7woaZXXNLjMHvsi7Yxqa1xhlyUqntHlGhXbLRJ4xSIdHlIyxVhZA0pRLBBEygNZCXUGDJrunPf6riFzHLTO0+a4VBE9BgxrVo54ssuaXFClrGKK93hRr9kucUGfba6S9rKEc1Y7Z8Ah9YoqihGDf05BtzYdTugUV8Iiy2UMyxmnIC0dKQwlFJETN8c9ck466FVNCs5Yb6O0o2Z61GQFVbrlrTZBXcRGk9Zut71KLnevNh0+YNgBVSZY6D2+aYkNHtejWVFGtRFtmby8hCmGzdDvVz7pJcd12OVLKprlVLnVsHqdfiyMWApH2M57hQ4ZMt1kKT0+pd9ZRxwR2OWIwHhzNctIRpovoxjFRSD29Ujzp9eAft1CPbZgmgE9SphnjrTpqgWR5lpCTiHSzRiWFXPIVv0aFS003nITnTbkLT3qpCWNN0FGLpowiuvSaIHLTfh/LWav4y61X9wszzsaMVYVVaJc7whfQoczTthvrisc84K1Muo9KW2cXa4VmmOMPo3KkfpQqE5Kd9QneKuSZhVVfudTOoQOyztltinanHZYMeL8DyIVj2FlQxErTtpZeYG4KoGURDRxUY40BkZmNGvMcUSfua4w3jZXabdTk/3S4uqdtFWDULcmQaS+MDIBcMQpLWqcUOcvnLdd4BbN8uq9hVNalSUjraaMfLR3aR83LGeyTRbZY6JzrvSwWYomaDAZ9xvWJqki8Y6WQlZZqy5nnXT+PzyLmfolXKKiwQT10ipyqkYtzCguDrFvRGpEeQVxWWklJ1HQr16N9DtaXcSjN19MWVIe8Uj7KysmJ+GCekNSsoblVWuMNDxqjGRERvSDilH9Iyk0ZNh4SUPGKCprltenIG9QLlI0q0R5g1LUGdPjuFY7o+mo/4jQu4wxwdgoc1OJMgdlJTEDjhnWYKbFik7ps9Jxp5w23RSBnEElMYlIQe5P7NxxRXkJI1w6I8wOYcQpESorR+f1T59CJa06lF1pvilC5xzwaSHOSivo8qgqtU5KvbP2EV27soJeGTXatBsvMKhDr/EmCAyoUSOuOmLMGfHrKhLGm2ehVMT5+//FXtXO2KVXURFlYaQPN6I9WVDSKS0jpcuAWvU61WjSraJatUp0h6MWZhQXhdj/iHRJgkgtrRhlSMlHJ62iJGGEC6+iGGk+liMmW5GOWFxJWRBpNZKMWPdHMsIpMUUpw5F/XhATRtqEZf+PPmvFiNUa0U8cUYNMRFqRucivp2JAXoMhcQUxp0wUSskJpQ1LSkdqhiQivSTvrLQoEekljiiuJgxFCpTxaA8qgkj/Nq6koFpOaDiqFQWRNkIyUmH9k22IR/qxxCKe7T/taZ0hoeV6XRBqFNOtQ41AV6RD9ScNx0Q0vx6LLHEoJ4x2JKUkJa8sGSkh/GmWaeRplCTVqhhCVtJBc5TkdGiWV6cSeZFh9BWTF0bPpaysJB7pRI6ow408pbhhBVXR/zVqYUZxkfi/ARx6SIYP9fz2AAAAAElFTkSuQmCC",
      "text/plain": [
       "28×280 Array{Gray{Float64},2} with eltype Gray{Float64}:\n",
       " Gray{Float64}(0.490824)  …  Gray{Float64}(0.500489)\n",
       " Gray{Float64}(0.509466)     Gray{Float64}(0.500019)\n",
       " Gray{Float64}(0.499039)     Gray{Float64}(0.490492)\n",
       " Gray{Float64}(0.508941)     Gray{Float64}(0.476173)\n",
       " Gray{Float64}(0.494655)     Gray{Float64}(0.488138)\n",
       " Gray{Float64}(0.508286)  …  Gray{Float64}(0.485166)\n",
       " Gray{Float64}(0.50035)      Gray{Float64}(0.464198)\n",
       " Gray{Float64}(0.485799)     Gray{Float64}(0.508036)\n",
       " Gray{Float64}(0.498516)     Gray{Float64}(0.475252)\n",
       " Gray{Float64}(0.490375)     Gray{Float64}(0.491196)\n",
       " Gray{Float64}(0.493878)  …  Gray{Float64}(0.498795)\n",
       " Gray{Float64}(0.510057)     Gray{Float64}(0.515459)\n",
       " Gray{Float64}(0.507569)     Gray{Float64}(0.510868)\n",
       " ⋮                        ⋱                         \n",
       " Gray{Float64}(0.493676)     Gray{Float64}(0.49869) \n",
       " Gray{Float64}(0.510784)     Gray{Float64}(0.54785) \n",
       " Gray{Float64}(0.513567)     Gray{Float64}(0.501581)\n",
       " Gray{Float64}(0.505647)     Gray{Float64}(0.490174)\n",
       " Gray{Float64}(0.506392)  …  Gray{Float64}(0.509558)\n",
       " Gray{Float64}(0.484479)     Gray{Float64}(0.491085)\n",
       " Gray{Float64}(0.509207)     Gray{Float64}(0.498947)\n",
       " Gray{Float64}(0.500104)     Gray{Float64}(0.494736)\n",
       " Gray{Float64}(0.514525)     Gray{Float64}(0.521575)\n",
       " Gray{Float64}(0.506117)  …  Gray{Float64}(0.514865)\n",
       " Gray{Float64}(0.509232)     Gray{Float64}(0.490231)\n",
       " Gray{Float64}(0.495311)     Gray{Float64}(0.487005)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "\"Epoch 99\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Let us visualize the evolution of the weight matrix as images below\n",
    "# Each row is turned into a 28x28 image with positive weights light and negative weights dark gray\n",
    "using Images, ImageMagick\n",
    "for t in 10 .^ range(0,stop=log10(size(lin,2)),length=20) #logspace(0,2,20)\n",
    "    i = ceil(Int,t)\n",
    "    f = lin[1,i]\n",
    "    w1 = reshape(Array(value(f.w))', (28,28,1,10))\n",
    "    w2 = clamp.(w1.+0.5,0,1)\n",
    "    IJulia.clear_output(true)\n",
    "    display(hcat([mnistview(w2,i) for i=1:10]...))\n",
    "    display(\"Epoch $(i-1)\")\n",
    "    sleep(1) # (0.96^i)\n",
    "end"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "julia.ipynb",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Julia 1.0.3",
   "language": "julia",
   "name": "julia-1.0"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.0.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
